Oct 30 18:35:48.431938 ip-10-0-133-96 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:35:48.897643 ip-10-0-133-96 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:35:48.897643 ip-10-0-133-96 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:35:48.897643 ip-10-0-133-96 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:35:48.897643 ip-10-0-133-96 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:35:48.902127 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902040 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:35:48.902127 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902126 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:35:48.902127 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902130 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902133 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902136 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902140 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902142 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902145 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902148 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902156 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902161 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902166 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902169 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902172 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902174 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902177 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902180 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902182 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902185 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902187 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902190 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:35:48.902220 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902192 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902195 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902197 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902200 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902202 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902205 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902207 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902210 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902212 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902214 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902217 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902219 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902222 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902224 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902227 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902230 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902233 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902235 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902239 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902242 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:35:48.902707 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902244 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902247 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902250 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902252 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902254 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902257 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902259 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902262 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902264 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902267 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902269 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902271 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902275 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902277 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902280 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902283 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902285 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902288 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902291 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:35:48.903163 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902294 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902296 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902298 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902301 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902303 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902306 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902308 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902311 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902313 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902316 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902318 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902321 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902324 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902327 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902329 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902332 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902335 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902337 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902340 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:35:48.903628 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902342 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902344 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902347 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902349 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902352 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902354 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902357 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902360 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902363 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902365 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902368 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902373 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902378 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902381 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902383 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902386 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902388 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902391 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902393 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:35:48.904106 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902397 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:35:48.904539 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902401 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:35:48.904539 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902403 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:35:48.904539 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902406 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:35:48.904539 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902408 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:35:48.904539 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902410 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:35:48.904539 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902413 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:35:48.904539 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.902416 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:35:48.905145 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905131 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:35:48.905145 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905145 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905148 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905151 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905154 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905157 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905160 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905163 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905165 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905168 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905171 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905175 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905177 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905180 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905182 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905185 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905187 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905190 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905194 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905198 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:35:48.905198 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905201 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905204 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905207 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905210 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905212 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905215 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905218 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905220 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905223 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905225 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905228 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905230 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905233 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905236 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905239 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905241 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905244 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905246 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905248 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:35:48.905678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905251 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905254 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905256 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905259 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905261 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905263 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905266 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905268 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905271 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905273 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905277 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905280 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905282 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905284 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905287 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905290 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905294 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905297 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905300 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905303 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:35:48.906133 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905305 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905308 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905310 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905313 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905315 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905318 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905321 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905323 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905326 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905329 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905341 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905344 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905347 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905350 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905352 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905355 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905357 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905360 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905362 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:35:48.906599 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905365 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905369 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905373 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905376 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905379 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905382 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905384 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905387 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905390 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905392 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905395 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905398 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905400 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905403 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905406 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905408 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905411 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905413 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905415 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:35:48.907051 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905418 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905421 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905424 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905426 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905429 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905431 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905434 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905436 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905439 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905955 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905960 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905963 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905966 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905969 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905971 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905975 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905978 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905980 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905983 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905985 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:35:48.907519 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905988 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905992 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905995 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.905997 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906000 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906002 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906005 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906008 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906010 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906013 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906016 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906018 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906021 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906024 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906026 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906029 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906031 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906033 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906036 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:35:48.908011 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906040 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906043 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906046 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906049 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906052 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906055 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906057 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906060 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906063 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906066 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906068 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906071 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906073 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906076 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906079 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906082 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906087 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906090 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906094 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:35:48.908458 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906097 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906100 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906102 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906105 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906108 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906111 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906113 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906129 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906134 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906138 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906141 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906144 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906146 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906149 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906151 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906154 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906156 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906159 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906162 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906165 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:35:48.908997 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906168 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906170 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906173 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906175 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906179 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906182 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906184 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906187 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906190 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906193 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906195 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906198 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906201 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906203 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906206 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906208 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906210 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906213 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906216 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:35:48.909472 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906218 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906221 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906224 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906226 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906229 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906231 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906234 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906236 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906239 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906241 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906244 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906246 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906249 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906252 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906254 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906257 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906260 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.906262 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907299 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907312 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907319 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:35:48.909980 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907324 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907329 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907332 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907336 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907340 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907344 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907347 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907351 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907354 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907357 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907360 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907363 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907366 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907369 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907373 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907378 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907382 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907385 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907388 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907392 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907402 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907405 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907408 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:35:48.910464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907412 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907415 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907419 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907422 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907425 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907428 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907433 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907436 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907439 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907442 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907445 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907448 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907452 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907456 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907458 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907462 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907465 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907469 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907472 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907475 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907478 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907481 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907484 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907487 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907490 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:35:48.911042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907493 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907496 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907499 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907503 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907506 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907509 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907512 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907515 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907521 2573 flags.go:64] FLAG: --help="false" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907524 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907528 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907531 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907534 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907538 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907541 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907544 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907547 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907550 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907553 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907556 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907559 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907562 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907565 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907568 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:35:48.911691 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907571 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907574 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907577 2573 flags.go:64] FLAG: --lock-file="" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907580 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907583 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907586 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907592 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907595 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907598 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907613 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907616 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907620 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907622 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907625 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907631 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907634 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907640 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907643 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907646 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907649 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907652 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907655 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907658 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907661 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907669 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:35:48.912261 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907672 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907675 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907679 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907682 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907685 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907688 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907691 2573 flags.go:64] FLAG: --port="10250" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907694 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907696 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08f6538b272b50780" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907699 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907702 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907705 2573 flags.go:64] FLAG: --register-node="true" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907708 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907712 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907715 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907717 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907721 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907725 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907728 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907731 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907734 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907737 2573 flags.go:64] FLAG: --runonce="false" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907740 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907744 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907747 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907750 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907753 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:35:48.912857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907756 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907759 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907762 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907765 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907768 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907771 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907774 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907777 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907780 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907783 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907788 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907791 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907794 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907799 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907802 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907804 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907807 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907811 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907813 2573 flags.go:64] FLAG: --v="2" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907818 2573 flags.go:64] FLAG: --version="false" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907822 2573 flags.go:64] FLAG: --vmodule="" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907826 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.907830 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.907989 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:35:48.913495 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.907992 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.907996 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.907999 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908002 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908006 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908013 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908015 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908018 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908021 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908024 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908026 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908029 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908032 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908034 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908037 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908039 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908042 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908044 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908047 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908049 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:35:48.914081 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908052 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908054 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908056 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908059 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908062 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908065 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908067 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908070 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908073 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908075 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908078 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908082 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908085 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908087 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908090 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908092 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908096 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908099 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908103 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:35:48.914590 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908106 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908109 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908112 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908114 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908117 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908119 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908122 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908124 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908126 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908129 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908131 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908133 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908136 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908138 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908140 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908143 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908146 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908148 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908150 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908153 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:35:48.915057 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908155 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908158 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908160 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908163 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908165 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908169 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908171 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908174 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908176 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908180 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908184 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908187 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908189 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908192 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908194 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908197 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908200 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908203 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908205 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:35:48.915541 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908207 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908210 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908212 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908215 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908217 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908220 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908222 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908225 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908227 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908230 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908232 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908235 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908237 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908240 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908242 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908245 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908247 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908249 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908252 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908258 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:35:48.916016 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908261 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908265 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908270 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908274 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908277 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908279 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908282 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.908312 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908354 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908360 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908363 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908366 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908368 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908371 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908373 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:35:48.916486 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908376 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908378 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908381 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908384 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908386 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908389 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908391 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908393 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908396 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908398 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908401 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908403 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908406 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908408 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908411 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908413 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908416 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908419 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908422 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:35:48.916863 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908428 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908430 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908434 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908437 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908440 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908442 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908445 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908447 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908450 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908452 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908455 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908457 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908460 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908462 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908465 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908467 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908470 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908473 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908475 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908478 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:35:48.917310 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908480 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908482 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908485 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908487 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908489 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908492 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908494 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908497 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908499 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908502 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908505 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908508 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908511 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908514 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908517 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908520 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908524 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908526 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908529 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908532 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:35:48.917813 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908534 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908537 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908539 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908542 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908545 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908549 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908552 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908555 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908557 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908560 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908562 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908564 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908567 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908569 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908572 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908574 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908576 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908579 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908581 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:35:48.918280 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908583 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908586 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908588 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908591 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908593 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908596 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908599 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908615 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908617 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908620 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908623 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908625 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908628 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908631 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908633 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908636 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908639 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908641 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908643 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:35:48.918732 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908647 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.908650 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.915942 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.915961 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.916737 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.916743 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916804 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916811 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916815 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916819 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916823 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916827 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916830 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916833 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916835 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916838 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916841 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916843 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916846 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916848 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:35:48.919177 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916851 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916853 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916856 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916859 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916861 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916864 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916867 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916869 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916872 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916875 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916878 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916881 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916883 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916886 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916889 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916893 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916895 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916898 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916900 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916903 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:35:48.919654 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916905 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916908 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916911 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916913 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916916 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916918 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916921 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916923 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916926 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916928 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916931 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916933 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916936 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916940 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916944 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916947 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916949 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916952 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916954 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:35:48.920156 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916957 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916959 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916962 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916964 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916967 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916970 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916973 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916975 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916978 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916981 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916984 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916986 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916989 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916991 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916993 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916996 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.916999 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917001 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917003 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:35:48.920612 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917006 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917008 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917011 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917013 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917015 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917018 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917020 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917022 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917025 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917028 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917030 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917033 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917035 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917037 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917040 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917042 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917045 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917048 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917051 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917053 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:35:48.921304 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917056 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917058 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917060 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917064 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917066 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917069 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917071 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917074 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917076 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917079 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917081 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917084 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917087 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917089 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.917122 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917295 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:35:48.922047 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917300 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917304 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917307 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917310 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917313 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917315 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917318 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917321 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917324 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917326 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917328 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917340 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917343 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917346 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917349 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917351 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917355 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917359 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917362 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:35:48.922464 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917365 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917369 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917372 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917374 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917377 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917379 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917382 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917384 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917387 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917389 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917392 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917394 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917397 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917400 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917402 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917405 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917408 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917411 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917413 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:35:48.923021 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917416 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917419 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917421 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917423 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917426 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917428 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917431 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917434 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917437 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917439 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917442 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917444 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917447 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917449 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917452 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917455 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917457 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917460 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917463 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:35:48.923466 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917467 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917470 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917473 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917475 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917478 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917480 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917483 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917486 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917488 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917491 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917493 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917496 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917498 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917501 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917503 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917505 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917508 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917510 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917513 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917515 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:35:48.923960 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917518 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917520 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917523 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917526 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917528 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917531 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917533 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917536 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917538 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917542 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917544 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917547 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917549 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917552 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917554 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917557 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917559 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917562 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917564 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917567 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:35:48.924438 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917570 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917572 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917575 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917577 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917579 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917582 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917584 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917587 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.917630 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917667 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917671 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917674 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917677 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917681 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917683 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917686 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:35:48.924920 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917689 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917692 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917695 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917698 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917701 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917703 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917706 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917709 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917712 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917714 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917717 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917719 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917722 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917726 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917729 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917732 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917734 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917737 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917739 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:35:48.925289 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917742 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917744 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917747 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917749 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917752 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917754 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917757 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917760 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917762 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917765 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917767 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917770 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917773 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917775 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917777 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917780 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917786 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917789 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917791 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:35:48.925753 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917794 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917797 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917799 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917802 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917804 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917806 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917809 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917811 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917814 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917816 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917819 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917822 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917824 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917827 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917831 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917834 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917838 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917841 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917843 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917847 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:35:48.926200 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917850 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917854 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917857 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917860 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917862 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917866 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917868 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917871 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917873 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917876 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917879 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917881 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917883 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917886 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917889 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917891 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917894 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917896 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917899 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917901 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:35:48.926846 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917904 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917906 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917909 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917911 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917914 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917916 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917918 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917921 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917923 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917926 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917928 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917931 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917934 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917937 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917939 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917942 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917944 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917948 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917950 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917952 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:35:48.927332 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:48.917955 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:35:48.927811 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.918012 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:35:48.927811 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.920553 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 18:35:48.927811 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.922318 2573 server.go:1014] "Starting client certificate rotation" Oct 30 18:35:48.927811 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.922429 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:35:48.927811 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.922471 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:35:48.927811 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.924591 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:35:48.927811 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.925749 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:35:48.935876 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.935855 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:35:48.941476 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.941454 2573 log.go:25] "Validated CRI v1 image API" Oct 30 18:35:48.942389 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.942358 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:35:48.945160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.945135 2573 fs.go:135] Filesystem UUIDs: map[256dd621-7796-4e21-8cce-9866d8d51c19:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a7af5301-f3e2-4eb8-9d5f-dc0e5ab8e9f5:/dev/nvme0n1p4] Oct 30 18:35:48.945160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.945155 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 18:35:48.949523 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.949399 2573 manager.go:217] Machine: {Timestamp:2026-10-30 18:35:48.94887988 +0000 UTC m=+0.384381119 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099506 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e3f93cf02172a46e97bf96b78a7b SystemUUID:ec21e3f9-3cf0-2172-a46e-97bf96b78a7b BootID:9e798b80-585c-4be9-a285-03490c834408 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:ae:05:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:ae:05:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:9a:69:1d:92:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:35:48.949523 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.949520 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:35:48.949650 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.949616 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:35:48.953145 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.953120 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:35:48.953303 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.953156 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-96.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:35:48.953356 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.953305 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:35:48.953356 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.953313 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:35:48.953356 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.953342 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:35:48.953432 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.953361 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:35:48.954548 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.954537 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:35:48.957190 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.957172 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:35:48.959544 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.959532 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:35:48.959696 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.959687 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:35:48.959731 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.959723 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:35:48.959759 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.959742 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:35:48.959759 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.959756 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:35:48.959824 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.959764 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:35:48.959824 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.959772 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:35:48.960357 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.960344 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:35:48.964206 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.964182 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:35:48.971929 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.971911 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:35:48.972847 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.972834 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:35:48.972887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.972866 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:35:48.973029 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973018 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:35:48.973064 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973035 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:35:48.973064 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973041 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:35:48.973064 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973047 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:35:48.973064 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973052 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:35:48.973064 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973060 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:35:48.973839 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973829 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:35:48.973839 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973839 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:35:48.973902 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973846 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:35:48.973902 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973853 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:35:48.973902 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973861 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:35:48.973902 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.973871 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:35:48.974754 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.974745 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:35:48.974754 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.974755 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:35:48.975536 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.975519 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6jrj" Oct 30 18:35:48.978859 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.978846 2573 server.go:1263] "Started kubelet" Oct 30 18:35:48.978983 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.978936 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:35:48.979019 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.978950 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:35:48.979019 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.979012 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:35:48.979639 ip-10-0-133-96 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:35:48.980555 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.980533 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:35:48.982320 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.982300 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:35:48.983898 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:48.983880 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 18:35:48.984326 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.984310 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6jrj" Oct 30 18:35:48.985224 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.985206 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-96.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 18:35:48.985300 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:48.985215 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-96.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 18:35:48.985389 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:48.985365 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 18:35:48.986292 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:48.985146 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-96.us-east-2.compute.internal.18e362791f45abd8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-96.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-96.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:35:48.978813912 +0000 UTC m=+0.414315151,LastTimestamp:2026-10-30 18:35:48.978813912 +0000 UTC m=+0.414315151,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-96.us-east-2.compute.internal,}" Oct 30 18:35:48.988889 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.988873 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:35:48.989448 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.989433 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:35:48.989616 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:48.989578 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:48.990178 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.990161 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:35:48.990178 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.990167 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:35:48.990306 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.990184 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:35:48.990306 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.990300 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:35:48.990306 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.990306 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:35:48.993466 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.993449 2573 factory.go:55] Registering systemd factory Oct 30 18:35:48.993550 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.993470 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:35:48.993755 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.993739 2573 factory.go:153] Registering CRI-O factory Oct 30 18:35:48.993818 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.993760 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 18:35:48.993985 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.993959 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:35:48.994020 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.994001 2573 factory.go:103] Registering Raw factory Oct 30 18:35:48.994020 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.994017 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 18:35:48.995321 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.995304 2573 manager.go:319] Starting recovery of all containers Oct 30 18:35:48.999806 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:48.999780 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:35:49.002358 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.002331 2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-96.us-east-2.compute.internal\" not found" node="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.007933 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.007917 2573 manager.go:324] Recovery completed Oct 30 18:35:49.011938 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.011926 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:35:49.013893 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.013875 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:35:49.013966 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.013909 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:35:49.013966 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.013926 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:35:49.014459 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.014443 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:35:49.014459 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.014457 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:35:49.014581 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.014474 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:35:49.014581 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.014445 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:35:49.015232 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.015216 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:35:49.015298 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.015250 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:35:49.015298 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.015265 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:35:49.016677 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.016664 2573 policy_none.go:50] "Start" Oct 30 18:35:49.016722 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.016681 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:35:49.016722 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.016695 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:35:49.018774 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.018763 2573 policy_none.go:44] "Start" Oct 30 18:35:49.070739 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.070534 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:35:49.075569 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.075550 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 18:35:49.075661 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.075621 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:35:49.075661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.075632 2573 server.go:87] "Starting device plugin registration server" Oct 30 18:35:49.075899 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.075888 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:35:49.075946 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.075900 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:35:49.076014 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.075996 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:35:49.076214 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.076094 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:35:49.076214 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.076110 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:35:49.076759 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.076740 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:35:49.076854 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.076791 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.134527 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.134477 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:35:49.136682 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.135873 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:35:49.136682 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.135896 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:35:49.136682 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.135922 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:35:49.136682 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.135957 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 18:35:49.142396 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.142374 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:35:49.176541 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.176464 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:35:49.177541 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.177524 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:35:49.177680 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.177559 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:35:49.177680 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.177575 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:35:49.177680 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.177616 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.185897 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.185880 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.186036 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.185905 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-96.us-east-2.compute.internal\": node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.206204 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.206179 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.236322 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.236271 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal"] Oct 30 18:35:49.236438 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.236372 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:35:49.237405 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.237389 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:35:49.237478 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.237419 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:35:49.237478 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.237434 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:35:49.239778 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.239767 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:35:49.239925 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.239911 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.239963 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.239941 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:35:49.240553 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.240536 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:35:49.240553 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.240551 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:35:49.240725 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.240570 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:35:49.240725 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.240586 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:35:49.240725 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.240571 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:35:49.240725 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.240625 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:35:49.242777 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.242761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.242840 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.242789 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:35:49.243471 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.243456 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:35:49.243542 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.243485 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:35:49.243542 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.243494 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:35:49.264330 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.264302 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" node="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.268417 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.268394 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" node="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.292009 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.291973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.292009 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.292008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.292200 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.292033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8517d60336db91a5d4c1afa2b83dc77-config\") pod \"kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"e8517d60336db91a5d4c1afa2b83dc77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.307339 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.307301 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.392565 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.392528 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8517d60336db91a5d4c1afa2b83dc77-config\") pod \"kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"e8517d60336db91a5d4c1afa2b83dc77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.392762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.392526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8517d60336db91a5d4c1afa2b83dc77-config\") pod \"kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"e8517d60336db91a5d4c1afa2b83dc77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.392762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.392624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.392762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.392674 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.392762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.392653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.392762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.392709 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.407665 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.407625 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.508539 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.508475 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.567710 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.567684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.571461 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.571433 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.609622 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.609570 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.710159 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.710118 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.810733 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.810654 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.911212 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.911182 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:35:49.922340 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.922310 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 18:35:49.922499 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.922479 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:35:49.922568 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.922513 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:35:49.946507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.946479 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:35:49.960574 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.960536 2573 apiserver.go:52] "Watching apiserver" Oct 30 18:35:49.971633 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.970441 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:35:49.971633 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.971123 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-vw4fs","openshift-image-registry/node-ca-slltd","openshift-multus/multus-dbklw","openshift-multus/network-metrics-daemon-qj49w","kube-system/konnectivity-agent-g6tw4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj","openshift-multus/multus-additional-cni-plugins-snt75","openshift-network-diagnostics/network-check-target-cbh4k","openshift-ovn-kubernetes/ovnkube-node-6lwpq","openshift-dns/node-resolver-msdfn","openshift-network-operator/iptables-alerter-fdhrx","kube-system/global-pull-secret-syncer-628cx"] Oct 30 18:35:49.973783 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.973748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:49.973910 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.973839 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:35:49.975935 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.975912 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.978046 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.978026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.980205 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.980184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:49.982364 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.982346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.983766 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.983737 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:35:49.983766 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.983759 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:35:49.983910 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.983793 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:35:49.983910 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.983795 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:35:49.983910 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.983812 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:35:49.983910 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.983799 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:35:49.983910 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.983841 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ldjxj\"" Oct 30 18:35:49.984514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.984502 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:49.985451 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985434 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:35:49.985536 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985434 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8j87p\"" Oct 30 18:35:49.985536 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985495 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:35:49.985536 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985502 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:35:49.985738 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985558 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:35:49.985738 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985563 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lfvmh\"" Oct 30 18:35:49.985738 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985619 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:35:49.985738 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985696 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:35:49.985874 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985745 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:35:49.985874 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985759 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:35:49.985874 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.985761 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:35:49.986680 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.986667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:49.987136 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.987123 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s4hwp\"" Oct 30 18:35:49.988854 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.988842 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:49.989016 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.989001 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 18:35:49.989901 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.989889 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:49.990421 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.990406 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:35:49.991072 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.991057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:49.991137 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.991119 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:35:49.992058 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.992020 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:30:49 +0000 UTC" deadline="2028-07-22 00:44:57.1417467 +0000 UTC" Oct 30 18:35:49.992058 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.992058 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15126h9m7.149691346s" Oct 30 18:35:49.993715 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.993685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:49.995622 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-etc-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.995690 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-netd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.995690 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-script-lib\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.995773 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-netns\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.995773 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995759 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-multus-certs\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.995856 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-bin\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.995856 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995830 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-lib-modules\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.995856 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-host\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.995992 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-tmp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.995992 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-socket-dir-parent\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.995992 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.995978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-socket-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:49.996085 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-system-cni-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:49.996085 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-ovn\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.996085 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-netns\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.996085 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:49.996244 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-node-log\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.996244 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996115 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:49.996244 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:49.996244 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-log-socket\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.996244 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996201 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw9k9\" (UniqueName: \"kubernetes.io/projected/a21962c6-ef9b-413f-abe8-a93835acebbc-kube-api-access-jw9k9\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.996244 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-modprobe-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.996460 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/272dcf18-8186-45d9-9316-8c5a0efc951b-serviceca\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:49.996460 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-conf-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.996460 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btts9\" (UniqueName: \"kubernetes.io/projected/871c9271-d966-4c0d-b367-be51d1d4a79d-kube-api-access-btts9\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:49.996460 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-hostroot\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.996460 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:49.996460 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996435 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-config\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.996717 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996492 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:49.996774 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-tuned\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.996824 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996773 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/272dcf18-8186-45d9-9316-8c5a0efc951b-host\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:49.996824 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdpg\" (UniqueName: \"kubernetes.io/projected/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-kube-api-access-nwdpg\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.996907 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-kubelet\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.996907 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-slash\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.996907 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.997049 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.997049 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-konnectivity-ca\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:49.997049 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.996990 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-registration-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:49.997049 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxh5h\" (UniqueName: \"kubernetes.io/projected/cc878e8c-7351-4591-9939-4d33d63b6d1e-kube-api-access-zxh5h\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:49.997289 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-env-overrides\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.997289 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysconfig\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.997289 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-run\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.997289 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-kubelet\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997289 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.997289 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997289 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cnibin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997321 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cni-binary-copy\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:49.997507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.997507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-kubernetes\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.997507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-sys\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.997507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nvxp\" (UniqueName: \"kubernetes.io/projected/f2b77242-e22d-4689-9fd6-545ef1da88c4-kube-api-access-9nvxp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.997507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:49.997762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-systemd-units\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.997762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997558 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-var-lib-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.997762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-bin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997630 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-etc-kubernetes\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-agent-certs\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:49.997762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a21962c6-ef9b-413f-abe8-a93835acebbc-ovn-node-metrics-cert\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.997762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-conf\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.997762 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmql\" (UniqueName: \"kubernetes.io/projected/272dcf18-8186-45d9-9316-8c5a0efc951b-kube-api-access-qhmql\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:49.997994 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997774 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-k8s-cni-cncf-io\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997994 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-daemon-config\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997994 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-device-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:49.997994 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-cnibin\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:49.997994 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n855\" (UniqueName: \"kubernetes.io/projected/89a132e7-7020-4c9e-accd-89b0a61d73b0-kube-api-access-5n855\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:49.997994 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-system-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.997994 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.997972 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:49.998313 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.998015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-sys-fs\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:49.998313 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.998072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:49.998313 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.998110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-systemd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:49.998313 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.998142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-systemd\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.998313 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.998160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-var-lib-kubelet\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:49.998313 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.998186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-multus\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:49.999059 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:49.999044 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:49.999112 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:49.999100 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:35:50.000862 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.000845 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal"] Oct 30 18:35:50.000931 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.000913 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:35:50.000973 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.000963 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:35:50.011618 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.011568 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal"] Oct 30 18:35:50.011851 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.011801 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:35:50.056015 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.055986 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:35:50.056210 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.056061 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2zsrb" Oct 30 18:35:50.061711 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.061568 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2zsrb" Oct 30 18:35:50.084315 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.084286 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:35:50.085871 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.085854 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:35:50.086670 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.086325 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:35:50.086670 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.086438 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:35:50.086670 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.086456 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:35:50.086670 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.086460 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:35:50.086670 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.086634 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:35:50.087076 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.087033 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:35:50.087174 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.087115 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:35:50.087229 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.087204 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89nks\"" Oct 30 18:35:50.087987 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.087950 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sfbzk\"" Oct 30 18:35:50.088081 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.088040 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xr42l\"" Oct 30 18:35:50.088081 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.088080 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjz7h\"" Oct 30 18:35:50.088236 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.088196 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8njxf\"" Oct 30 18:35:50.090797 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.090780 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:35:50.099103 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-dbus\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.099215 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-node-log\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.099215 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.099330 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:50.099330 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-log-socket\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.099330 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw9k9\" (UniqueName: \"kubernetes.io/projected/a21962c6-ef9b-413f-abe8-a93835acebbc-kube-api-access-jw9k9\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.099464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-modprobe-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.099464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/272dcf18-8186-45d9-9316-8c5a0efc951b-serviceca\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:50.099464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099376 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-log-socket\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.099464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-conf-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.099464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099412 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.099464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099335 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-node-log\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.099464 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btts9\" (UniqueName: \"kubernetes.io/projected/871c9271-d966-4c0d-b367-be51d1d4a79d-kube-api-access-btts9\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:50.099887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-hostroot\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.099887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.099887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-iptables-alerter-script\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.099887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-config\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.099887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-tuned\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.099887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/272dcf18-8186-45d9-9316-8c5a0efc951b-host\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:50.099887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099807 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-modprobe-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.099887 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.099880 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/272dcf18-8186-45d9-9316-8c5a0efc951b-serviceca\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:50.100264 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-conf-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.100264 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100070 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-hostroot\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.100264 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdpg\" (UniqueName: \"kubernetes.io/projected/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-kube-api-access-nwdpg\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.100264 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.100264 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100218 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/272dcf18-8186-45d9-9316-8c5a0efc951b-host\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:50.100264 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100216 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100257 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-kubelet\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-kubelet\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-slash\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-konnectivity-ca\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100430 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-slash\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-registration-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxh5h\" (UniqueName: \"kubernetes.io/projected/cc878e8c-7351-4591-9939-4d33d63b6d1e-kube-api-access-zxh5h\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8994e33-9b53-41d7-8fa8-04463a3ae647-tmp-dir\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.100547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-env-overrides\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.100931 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100560 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.100931 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100638 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.100931 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100649 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysconfig\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.100931 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100636 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-registration-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.100931 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysconfig\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.100931 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100670 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-config\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.101628 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101586 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-env-overrides\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.101735 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-run\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.101735 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-konnectivity-ca\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:50.101735 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-kubelet\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.101888 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101741 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-kubelet\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.101888 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5zg5\" (UniqueName: \"kubernetes.io/projected/c8994e33-9b53-41d7-8fa8-04463a3ae647-kube-api-access-q5zg5\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.101888 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-host-slash\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.101888 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.101888 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.101888 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cnibin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.101888 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101886 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-run\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.102119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cni-binary-copy\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.102119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.102119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.102119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.101983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-kubernetes\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.102119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-sys\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.102119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nvxp\" (UniqueName: \"kubernetes.io/projected/f2b77242-e22d-4689-9fd6-545ef1da88c4-kube-api-access-9nvxp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.102119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.102119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8994e33-9b53-41d7-8fa8-04463a3ae647-hosts-file\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.102379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102121 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cnibin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.102379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-systemd-units\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.102379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-var-lib-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.102379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-bin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.102379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-etc-kubernetes\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.102379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-agent-certs\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:50.102379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-kubelet-config\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a21962c6-ef9b-413f-abe8-a93835acebbc-ovn-node-metrics-cert\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-conf\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.100473 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmql\" (UniqueName: \"kubernetes.io/projected/272dcf18-8186-45d9-9316-8c5a0efc951b-kube-api-access-qhmql\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102531 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-systemd-units\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102546 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-sys\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-k8s-cni-cncf-io\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-daemon-config\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.102730 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102710 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-device-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-cnibin\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102825 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-kubernetes\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102912 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-device-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n855\" (UniqueName: \"kubernetes.io/projected/89a132e7-7020-4c9e-accd-89b0a61d73b0-kube-api-access-5n855\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102971 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-var-lib-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-system-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103118 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-conf\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.103170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103148 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-sys-fs\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103204 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103251 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-cnibin\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103289 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-k8s-cni-cncf-io\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-systemd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-systemd\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-var-lib-kubelet\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103384 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-system-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.102495 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-etc-kubernetes\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103333 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103446 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-systemd\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103479 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-var-lib-kubelet\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.103564 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103543 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-bin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103643 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-sys-fs\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103765 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-multus\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-etc-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-netd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-script-lib\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103918 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-systemd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-netns\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-multus-certs\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.103992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-bin\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104013 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-lib-modules\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-host\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104090 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-multus\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-tmp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104137 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-bin\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104140 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104096 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-lib-modules\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-socket-dir-parent\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-socket-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104212 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-socket-dir-parent\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-system-cni-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-system-cni-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-ovn\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104292 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-ovn\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-netns\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104333 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-netns\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwjc\" (UniqueName: \"kubernetes.io/projected/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-kube-api-access-chwjc\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104388 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-netns\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104515 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-socket-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.104645 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.104728 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs podName:871c9271-d966-4c0d-b367-be51d1d4a79d nodeName:}" failed. No retries permitted until 2026-10-30 18:35:50.604687369 +0000 UTC m=+2.040188613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs") pod "network-metrics-daemon-qj49w" (UID: "871c9271-d966-4c0d-b367-be51d1d4a79d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:50.104842 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104723 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cni-binary-copy\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104978 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-multus-certs\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.105025 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-etc-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.105032 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-daemon-config\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.105403 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.105407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.105526 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.104035 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-netd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.105547 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-tuned\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.105802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.105630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-host\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.106866 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.106623 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-script-lib\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.107843 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.107823 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-tmp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.108439 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.108411 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a21962c6-ef9b-413f-abe8-a93835acebbc-ovn-node-metrics-cert\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.108797 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.108779 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-agent-certs\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:50.122025 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.122001 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:35:50.122025 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.122023 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:35:50.122025 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.122034 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bv2pn for pod openshift-network-diagnostics/network-check-target-cbh4k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:50.122237 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.122083 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn podName:b52d343a-4ee8-474d-b2ab-6fc96edde2fd nodeName:}" failed. No retries permitted until 2026-10-30 18:35:50.622067717 +0000 UTC m=+2.057568942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bv2pn" (UniqueName: "kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn") pod "network-check-target-cbh4k" (UID: "b52d343a-4ee8-474d-b2ab-6fc96edde2fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:50.123227 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.123192 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxh5h\" (UniqueName: \"kubernetes.io/projected/cc878e8c-7351-4591-9939-4d33d63b6d1e-kube-api-access-zxh5h\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.123342 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.123288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nvxp\" (UniqueName: \"kubernetes.io/projected/f2b77242-e22d-4689-9fd6-545ef1da88c4-kube-api-access-9nvxp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.124879 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.124856 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n855\" (UniqueName: \"kubernetes.io/projected/89a132e7-7020-4c9e-accd-89b0a61d73b0-kube-api-access-5n855\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.125280 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.125259 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmql\" (UniqueName: \"kubernetes.io/projected/272dcf18-8186-45d9-9316-8c5a0efc951b-kube-api-access-qhmql\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:50.125477 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.125457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw9k9\" (UniqueName: \"kubernetes.io/projected/a21962c6-ef9b-413f-abe8-a93835acebbc-kube-api-access-jw9k9\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.125678 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.125659 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btts9\" (UniqueName: \"kubernetes.io/projected/871c9271-d966-4c0d-b367-be51d1d4a79d-kube-api-access-btts9\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:50.134355 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.134322 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdpg\" (UniqueName: \"kubernetes.io/projected/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-kube-api-access-nwdpg\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.189967 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.189931 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8517d60336db91a5d4c1afa2b83dc77.slice/crio-be0307adac5f8682c13f30b6c44ae7df5217989fe5a8d87c1a9ee18ccc4ca2d1 WatchSource:0}: Error finding container be0307adac5f8682c13f30b6c44ae7df5217989fe5a8d87c1a9ee18ccc4ca2d1: Status 404 returned error can't find the container with id be0307adac5f8682c13f30b6c44ae7df5217989fe5a8d87c1a9ee18ccc4ca2d1 Oct 30 18:35:50.191966 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.191943 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb42ec34eaffa661af422bced7d9962.slice/crio-d6e21717adb75fa3fdf56f43294b1cb5389c0a5ffa9280c7ce635560fedcb2a8 WatchSource:0}: Error finding container d6e21717adb75fa3fdf56f43294b1cb5389c0a5ffa9280c7ce635560fedcb2a8: Status 404 returned error can't find the container with id d6e21717adb75fa3fdf56f43294b1cb5389c0a5ffa9280c7ce635560fedcb2a8 Oct 30 18:35:50.194789 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.194770 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:35:50.205513 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.205640 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205521 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8994e33-9b53-41d7-8fa8-04463a3ae647-tmp-dir\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.205640 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zg5\" (UniqueName: \"kubernetes.io/projected/c8994e33-9b53-41d7-8fa8-04463a3ae647-kube-api-access-q5zg5\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.205753 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.205649 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:50.205753 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.205712 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret podName:7d77554d-8094-4546-8fe7-0edf66752a5e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:50.705692028 +0000 UTC m=+2.141193268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret") pod "global-pull-secret-syncer-628cx" (UID: "7d77554d-8094-4546-8fe7-0edf66752a5e") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:50.205753 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-host-slash\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.205906 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8994e33-9b53-41d7-8fa8-04463a3ae647-hosts-file\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.205906 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205806 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8994e33-9b53-41d7-8fa8-04463a3ae647-tmp-dir\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.205906 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-kubelet-config\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.205906 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205840 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8994e33-9b53-41d7-8fa8-04463a3ae647-hosts-file\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.205906 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205808 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-host-slash\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.205906 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205869 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-kubelet-config\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.205906 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwjc\" (UniqueName: \"kubernetes.io/projected/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-kube-api-access-chwjc\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.205906 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-dbus\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.206275 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-iptables-alerter-script\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.206275 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.205996 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-dbus\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.206459 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.206440 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-iptables-alerter-script\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.213724 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.213702 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zg5\" (UniqueName: \"kubernetes.io/projected/c8994e33-9b53-41d7-8fa8-04463a3ae647-kube-api-access-q5zg5\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.213814 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.213734 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwjc\" (UniqueName: \"kubernetes.io/projected/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-kube-api-access-chwjc\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.312982 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.312890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:35:50.316757 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.316727 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:35:50.320462 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.320394 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-slltd" Oct 30 18:35:50.323726 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.323701 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b77242_e22d_4689_9fd6_545ef1da88c4.slice/crio-cd4eeb5f6d17376dc70b41c7885f131b31f0522c90f4a096b34ebaf61c385b8e WatchSource:0}: Error finding container cd4eeb5f6d17376dc70b41c7885f131b31f0522c90f4a096b34ebaf61c385b8e: Status 404 returned error can't find the container with id cd4eeb5f6d17376dc70b41c7885f131b31f0522c90f4a096b34ebaf61c385b8e Oct 30 18:35:50.328099 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.328074 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21962c6_ef9b_413f_abe8_a93835acebbc.slice/crio-fa09d3e6f1b46069f3cef0db14329bda4be2d2952341a287404ce35edb72f55c WatchSource:0}: Error finding container fa09d3e6f1b46069f3cef0db14329bda4be2d2952341a287404ce35edb72f55c: Status 404 returned error can't find the container with id fa09d3e6f1b46069f3cef0db14329bda4be2d2952341a287404ce35edb72f55c Oct 30 18:35:50.331546 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.331523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dbklw" Oct 30 18:35:50.331785 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.331758 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272dcf18_8186_45d9_9316_8c5a0efc951b.slice/crio-508bd618fe005a261db3945088046ea1b6101b237b62bc1eb1db72e982437d00 WatchSource:0}: Error finding container 508bd618fe005a261db3945088046ea1b6101b237b62bc1eb1db72e982437d00: Status 404 returned error can't find the container with id 508bd618fe005a261db3945088046ea1b6101b237b62bc1eb1db72e982437d00 Oct 30 18:35:50.342304 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.342277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:50.342838 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.342812 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1378c8_31d1_4011_80ba_f5c0aad8e3d3.slice/crio-a0e139b70bb14fa05239c9166e60b22218a5eb30760ad8a4d832a8c726af2cf7 WatchSource:0}: Error finding container a0e139b70bb14fa05239c9166e60b22218a5eb30760ad8a4d832a8c726af2cf7: Status 404 returned error can't find the container with id a0e139b70bb14fa05239c9166e60b22218a5eb30760ad8a4d832a8c726af2cf7 Oct 30 18:35:50.351307 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.351286 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5ba25c_da20_40d5_b02f_89e94aaad0d2.slice/crio-6fbcb57cf18754b006c8d806fd1e4164b7234bc19cd69677f3841ceb9bee960e WatchSource:0}: Error finding container 6fbcb57cf18754b006c8d806fd1e4164b7234bc19cd69677f3841ceb9bee960e: Status 404 returned error can't find the container with id 6fbcb57cf18754b006c8d806fd1e4164b7234bc19cd69677f3841ceb9bee960e Oct 30 18:35:50.361230 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.361201 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:35:50.371379 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.371351 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc878e8c_7351_4591_9939_4d33d63b6d1e.slice/crio-662790f5602c3adfc5b440570e9c38e9e9690b27315e25f6bb06ba30956a47df WatchSource:0}: Error finding container 662790f5602c3adfc5b440570e9c38e9e9690b27315e25f6bb06ba30956a47df: Status 404 returned error can't find the container with id 662790f5602c3adfc5b440570e9c38e9e9690b27315e25f6bb06ba30956a47df Oct 30 18:35:50.389534 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.389499 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:35:50.392659 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.392637 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-msdfn" Oct 30 18:35:50.400721 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.400695 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:35:50.400980 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.400951 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a132e7_7020_4c9e_accd_89b0a61d73b0.slice/crio-a06c77803ed1a18115d666293b30bb15c5045a523b6b6ccdedae5ae879e71cde WatchSource:0}: Error finding container a06c77803ed1a18115d666293b30bb15c5045a523b6b6ccdedae5ae879e71cde: Status 404 returned error can't find the container with id a06c77803ed1a18115d666293b30bb15c5045a523b6b6ccdedae5ae879e71cde Oct 30 18:35:50.401578 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.401559 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8994e33_9b53_41d7_8fa8_04463a3ae647.slice/crio-04cdbf66861fa622868c7baf6998d9d45d6466ac731f195cb70655c86f6a3619 WatchSource:0}: Error finding container 04cdbf66861fa622868c7baf6998d9d45d6466ac731f195cb70655c86f6a3619: Status 404 returned error can't find the container with id 04cdbf66861fa622868c7baf6998d9d45d6466ac731f195cb70655c86f6a3619 Oct 30 18:35:50.409757 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:35:50.409735 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dc8719_07a1_41fe_834b_66bbd4e9e19b.slice/crio-a9e9f60d6c2931c6b6f0f7b6523a3f4e91f39eb6834d1edf73fe15007a760d1a WatchSource:0}: Error finding container a9e9f60d6c2931c6b6f0f7b6523a3f4e91f39eb6834d1edf73fe15007a760d1a: Status 404 returned error can't find the container with id a9e9f60d6c2931c6b6f0f7b6523a3f4e91f39eb6834d1edf73fe15007a760d1a Oct 30 18:35:50.453435 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.453411 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:35:50.609167 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.609080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:50.609312 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.609210 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:50.609312 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.609280 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs podName:871c9271-d966-4c0d-b367-be51d1d4a79d nodeName:}" failed. No retries permitted until 2026-10-30 18:35:51.609254602 +0000 UTC m=+3.044755829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs") pod "network-metrics-daemon-qj49w" (UID: "871c9271-d966-4c0d-b367-be51d1d4a79d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:50.710064 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.710017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:50.710064 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:50.710066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:50.710271 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.710186 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:50.710271 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.710233 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:35:50.710271 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.710265 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:35:50.710391 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.710280 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bv2pn for pod openshift-network-diagnostics/network-check-target-cbh4k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:50.710391 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.710247 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret podName:7d77554d-8094-4546-8fe7-0edf66752a5e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:51.710229533 +0000 UTC m=+3.145730776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret") pod "global-pull-secret-syncer-628cx" (UID: "7d77554d-8094-4546-8fe7-0edf66752a5e") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:50.710391 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:50.710353 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn podName:b52d343a-4ee8-474d-b2ab-6fc96edde2fd nodeName:}" failed. No retries permitted until 2026-10-30 18:35:51.710335349 +0000 UTC m=+3.145836593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bv2pn" (UniqueName: "kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn") pod "network-check-target-cbh4k" (UID: "b52d343a-4ee8-474d-b2ab-6fc96edde2fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:51.065568 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.065446 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:50 +0000 UTC" deadline="2028-06-27 16:16:13.667874597 +0000 UTC" Oct 30 18:35:51.065568 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.065515 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14541h40m22.602364835s" Oct 30 18:35:51.201486 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.201350 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6tw4" event={"ID":"ce5ba25c-da20-40d5-b02f-89e94aaad0d2","Type":"ContainerStarted","Data":"6fbcb57cf18754b006c8d806fd1e4164b7234bc19cd69677f3841ceb9bee960e"} Oct 30 18:35:51.209415 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.209373 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"e8517d60336db91a5d4c1afa2b83dc77","Type":"ContainerStarted","Data":"be0307adac5f8682c13f30b6c44ae7df5217989fe5a8d87c1a9ee18ccc4ca2d1"} Oct 30 18:35:51.231968 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.231892 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fdhrx" event={"ID":"d9dc8719-07a1-41fe-834b-66bbd4e9e19b","Type":"ContainerStarted","Data":"a9e9f60d6c2931c6b6f0f7b6523a3f4e91f39eb6834d1edf73fe15007a760d1a"} Oct 30 18:35:51.242102 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.242066 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-msdfn" event={"ID":"c8994e33-9b53-41d7-8fa8-04463a3ae647","Type":"ContainerStarted","Data":"04cdbf66861fa622868c7baf6998d9d45d6466ac731f195cb70655c86f6a3619"} Oct 30 18:35:51.258274 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.258225 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" event={"ID":"cc878e8c-7351-4591-9939-4d33d63b6d1e","Type":"ContainerStarted","Data":"662790f5602c3adfc5b440570e9c38e9e9690b27315e25f6bb06ba30956a47df"} Oct 30 18:35:51.273662 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.273621 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dbklw" event={"ID":"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3","Type":"ContainerStarted","Data":"a0e139b70bb14fa05239c9166e60b22218a5eb30760ad8a4d832a8c726af2cf7"} Oct 30 18:35:51.290045 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.289990 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-slltd" event={"ID":"272dcf18-8186-45d9-9316-8c5a0efc951b","Type":"ContainerStarted","Data":"508bd618fe005a261db3945088046ea1b6101b237b62bc1eb1db72e982437d00"} Oct 30 18:35:51.298183 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.298140 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"fa09d3e6f1b46069f3cef0db14329bda4be2d2952341a287404ce35edb72f55c"} Oct 30 18:35:51.301800 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.301771 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:35:51.310622 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.310581 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" event={"ID":"f2b77242-e22d-4689-9fd6-545ef1da88c4","Type":"ContainerStarted","Data":"cd4eeb5f6d17376dc70b41c7885f131b31f0522c90f4a096b34ebaf61c385b8e"} Oct 30 18:35:51.340759 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.340610 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"9fb42ec34eaffa661af422bced7d9962","Type":"ContainerStarted","Data":"d6e21717adb75fa3fdf56f43294b1cb5389c0a5ffa9280c7ce635560fedcb2a8"} Oct 30 18:35:51.361452 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.361381 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snt75" event={"ID":"89a132e7-7020-4c9e-accd-89b0a61d73b0","Type":"ContainerStarted","Data":"a06c77803ed1a18115d666293b30bb15c5045a523b6b6ccdedae5ae879e71cde"} Oct 30 18:35:51.469984 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.469944 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:35:51.618202 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.618115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:51.618385 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:51.618294 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:51.618385 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:51.618374 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs podName:871c9271-d966-4c0d-b367-be51d1d4a79d nodeName:}" failed. No retries permitted until 2026-10-30 18:35:53.618351877 +0000 UTC m=+5.053853124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs") pod "network-metrics-daemon-qj49w" (UID: "871c9271-d966-4c0d-b367-be51d1d4a79d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:51.718801 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.718762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:51.719012 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:51.718832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:51.719012 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:51.718971 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:51.719012 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:51.718983 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:35:51.719012 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:51.719013 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:35:51.719239 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:51.719029 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bv2pn for pod openshift-network-diagnostics/network-check-target-cbh4k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:51.719239 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:51.719033 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret podName:7d77554d-8094-4546-8fe7-0edf66752a5e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:53.719014815 +0000 UTC m=+5.154516068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret") pod "global-pull-secret-syncer-628cx" (UID: "7d77554d-8094-4546-8fe7-0edf66752a5e") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:51.719239 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:51.719087 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn podName:b52d343a-4ee8-474d-b2ab-6fc96edde2fd nodeName:}" failed. No retries permitted until 2026-10-30 18:35:53.719067406 +0000 UTC m=+5.154568641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bv2pn" (UniqueName: "kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn") pod "network-check-target-cbh4k" (UID: "b52d343a-4ee8-474d-b2ab-6fc96edde2fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:52.066521 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:52.066392 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:50 +0000 UTC" deadline="2028-04-30 18:59:16.010114477 +0000 UTC" Oct 30 18:35:52.066521 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:52.066469 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13152h23m23.943650831s" Oct 30 18:35:52.137049 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:52.137005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:52.137252 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:52.137165 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:35:52.137649 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:52.137623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:52.137814 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:52.137783 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:35:52.137910 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:52.137869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:52.137996 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:52.137948 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:35:53.637634 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:53.637589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:53.638232 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:53.637849 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:53.638232 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:53.637925 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs podName:871c9271-d966-4c0d-b367-be51d1d4a79d nodeName:}" failed. No retries permitted until 2026-10-30 18:35:57.637902364 +0000 UTC m=+9.073403614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs") pod "network-metrics-daemon-qj49w" (UID: "871c9271-d966-4c0d-b367-be51d1d4a79d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:53.738599 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:53.738553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:53.738823 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:53.738642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:53.738899 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:53.738816 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:53.738899 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:53.738890 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret podName:7d77554d-8094-4546-8fe7-0edf66752a5e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:57.738869476 +0000 UTC m=+9.174370710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret") pod "global-pull-secret-syncer-628cx" (UID: "7d77554d-8094-4546-8fe7-0edf66752a5e") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:53.739370 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:53.739352 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:35:53.739463 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:53.739379 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:35:53.739463 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:53.739392 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bv2pn for pod openshift-network-diagnostics/network-check-target-cbh4k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:53.739463 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:53.739445 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn podName:b52d343a-4ee8-474d-b2ab-6fc96edde2fd nodeName:}" failed. No retries permitted until 2026-10-30 18:35:57.739428652 +0000 UTC m=+9.174929886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bv2pn" (UniqueName: "kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn") pod "network-check-target-cbh4k" (UID: "b52d343a-4ee8-474d-b2ab-6fc96edde2fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:54.136700 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:54.136647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:54.136902 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:54.136798 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:35:54.137248 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:54.137222 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:54.137362 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:54.137340 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:35:54.137413 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:54.137399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:54.137496 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:54.137473 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:35:55.398426 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:55.398155 2573 generic.go:356] "Generic (PLEG): container finished" podID="9fb42ec34eaffa661af422bced7d9962" containerID="564ce27d691bee14ba879cf6090dee2d53a65bc9d723196cc0883d692e19ce06" exitCode=0 Oct 30 18:35:55.398960 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:55.398536 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"9fb42ec34eaffa661af422bced7d9962","Type":"ContainerDied","Data":"564ce27d691bee14ba879cf6090dee2d53a65bc9d723196cc0883d692e19ce06"} Oct 30 18:35:55.403395 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:55.403371 2573 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="c60d123982db97d47ce8ea9c2613dfcf5f9209eaefaf453d76997f6bf53eef1b" exitCode=0 Oct 30 18:35:55.403493 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:55.403415 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snt75" event={"ID":"89a132e7-7020-4c9e-accd-89b0a61d73b0","Type":"ContainerDied","Data":"c60d123982db97d47ce8ea9c2613dfcf5f9209eaefaf453d76997f6bf53eef1b"} Oct 30 18:35:56.137340 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:56.136334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:56.137340 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:56.136482 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:35:56.137340 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:56.136951 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:56.137340 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:56.137056 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:35:56.137340 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:56.137124 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:56.137340 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:56.137213 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:35:56.412874 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:56.412054 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"9fb42ec34eaffa661af422bced7d9962","Type":"ContainerStarted","Data":"bbd3429f5d3e4bd68c2d71e5093247a6072903538ac00a466e29ebd082cf5f90"} Oct 30 18:35:56.416360 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:56.415775 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6tw4" event={"ID":"ce5ba25c-da20-40d5-b02f-89e94aaad0d2","Type":"ContainerStarted","Data":"dc7f402bd9681da108788386d2c0f883ab254525c0dfffbdb3214b953683b004"} Oct 30 18:35:56.420871 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:56.420834 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"e8517d60336db91a5d4c1afa2b83dc77","Type":"ContainerStarted","Data":"049db31cc86f71141a5c2679dc6a59b6e83bec61d3e2694f37fa5f82f9cbf521"} Oct 30 18:35:56.447114 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:56.447042 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" podStartSLOduration=6.447020333 podStartE2EDuration="6.447020333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:56.429765725 +0000 UTC m=+7.865266980" watchObservedRunningTime="2026-10-30 18:35:56.447020333 +0000 UTC m=+7.882521585" Oct 30 18:35:56.462378 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:56.462320 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g6tw4" podStartSLOduration=1.49042947 podStartE2EDuration="6.462297067s" totalImagesPullingTime="4.971867597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:56.44774301 +0000 UTC m=+7.883244262" watchObservedRunningTime="2026-10-30 18:35:56.462297067 +0000 UTC m=+7.897798320" Oct 30 18:35:57.428630 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:57.428563 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-slltd" event={"ID":"272dcf18-8186-45d9-9316-8c5a0efc951b","Type":"ContainerStarted","Data":"13761452175ca5e3eb0c23868099d95b8907878fdad1ae44095e69fd458036b7"} Oct 30 18:35:57.445086 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:57.444992 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-slltd" podStartSLOduration=1.315361285 podStartE2EDuration="7.444971889s" totalImagesPullingTime="6.129610604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:57.444232463 +0000 UTC m=+8.879733716" watchObservedRunningTime="2026-10-30 18:35:57.444971889 +0000 UTC m=+8.880473144" Oct 30 18:35:57.445245 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:57.445172 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" podStartSLOduration=7.445158811 podStartE2EDuration="7.445158811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:56.462848498 +0000 UTC m=+7.898349751" watchObservedRunningTime="2026-10-30 18:35:57.445158811 +0000 UTC m=+8.880660067" Oct 30 18:35:57.673668 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:57.673604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:57.673873 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:57.673846 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:57.673960 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:57.673918 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs podName:871c9271-d966-4c0d-b367-be51d1d4a79d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:05.673897058 +0000 UTC m=+17.109398302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs") pod "network-metrics-daemon-qj49w" (UID: "871c9271-d966-4c0d-b367-be51d1d4a79d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:35:57.774520 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:57.774473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:57.774733 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:57.774590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:57.774806 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:57.774766 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:35:57.774806 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:57.774787 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:35:57.774806 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:57.774802 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bv2pn for pod openshift-network-diagnostics/network-check-target-cbh4k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:57.774967 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:57.774868 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn podName:b52d343a-4ee8-474d-b2ab-6fc96edde2fd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:05.774848711 +0000 UTC m=+17.210349944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bv2pn" (UniqueName: "kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn") pod "network-check-target-cbh4k" (UID: "b52d343a-4ee8-474d-b2ab-6fc96edde2fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:35:57.774967 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:57.774948 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:57.775081 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:57.774984 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret podName:7d77554d-8094-4546-8fe7-0edf66752a5e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:05.774971626 +0000 UTC m=+17.210472855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret") pod "global-pull-secret-syncer-628cx" (UID: "7d77554d-8094-4546-8fe7-0edf66752a5e") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:35:58.137306 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:58.137211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:35:58.137306 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:58.137267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:35:58.137557 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:58.137367 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:35:58.137557 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:58.137211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:35:58.137557 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:58.137472 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:35:58.137746 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:35:58.137582 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:35:58.441802 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:58.441661 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" event={"ID":"cc878e8c-7351-4591-9939-4d33d63b6d1e","Type":"ContainerStarted","Data":"a516926b50807d80775e99cbe0b4941759d9a4ee3d08d83bc0e772f54ddb07b9"} Oct 30 18:35:59.419117 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:59.419081 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:35:59.446369 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:59.446326 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" event={"ID":"cc878e8c-7351-4591-9939-4d33d63b6d1e","Type":"ContainerStarted","Data":"b7367c7b1cfca2f40a02b0636124153ce1701da4e8b89aa438a3f464d89542b2"} Oct 30 18:35:59.452170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:59.452127 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dbklw" event={"ID":"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3","Type":"ContainerStarted","Data":"292795fe45c862af888226230740911d7025d0de08d182d463ec669ce51b26a0"} Oct 30 18:35:59.746307 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:59.746215 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:59.748107 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:59.748074 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:35:59.764584 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:35:59.764524 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dbklw" podStartSLOduration=0.844027545 podStartE2EDuration="9.76450164s" totalImagesPullingTime="8.920474095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:59.469923075 +0000 UTC m=+10.905424327" watchObservedRunningTime="2026-10-30 18:35:59.76450164 +0000 UTC m=+11.200002894" Oct 30 18:36:00.083547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.083405 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:35:59.419111102Z","UUID":"4f1926aa-e04f-4b71-8550-802607004541","Handler":null,"Name":"","Endpoint":""} Oct 30 18:36:00.085146 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.085119 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:36:00.085258 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.085158 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:36:00.136899 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.136856 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:00.137079 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.136864 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:00.137079 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.137026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:00.137079 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:00.137019 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:00.137258 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:00.137150 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:00.137258 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:00.137205 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:00.458663 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.458475 2573 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="66b4d26a86e233b38599f4e6daeca14ad5ce4c84d5d5791ada1df12e84d2bb44" exitCode=0 Oct 30 18:36:00.458663 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.458581 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snt75" event={"ID":"89a132e7-7020-4c9e-accd-89b0a61d73b0","Type":"ContainerDied","Data":"66b4d26a86e233b38599f4e6daeca14ad5ce4c84d5d5791ada1df12e84d2bb44"} Oct 30 18:36:00.465431 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:00.464286 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-msdfn" event={"ID":"c8994e33-9b53-41d7-8fa8-04463a3ae647","Type":"ContainerStarted","Data":"faabb77d9eb09a726cd8713c55cb1d77f1880731ea1175dafb861e877c59d147"} Oct 30 18:36:01.468491 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:01.468278 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fdhrx" event={"ID":"d9dc8719-07a1-41fe-834b-66bbd4e9e19b","Type":"ContainerStarted","Data":"11a975cdfc6665ae726efc7f25a1e33fac7c49c4ce92546cdc61a15f87d4380e"} Oct 30 18:36:01.473125 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:01.473093 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" event={"ID":"cc878e8c-7351-4591-9939-4d33d63b6d1e","Type":"ContainerStarted","Data":"a3b913a257f74a081fec037fac6b81f3e505106612eeccd5a292e3d30df8f8d5"} Oct 30 18:36:01.483741 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:01.483674 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-msdfn" podStartSLOduration=2.42889912 podStartE2EDuration="11.483655182s" totalImagesPullingTime="9.054756062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:00.497659973 +0000 UTC m=+11.933161221" watchObservedRunningTime="2026-10-30 18:36:01.483655182 +0000 UTC m=+12.919156431" Oct 30 18:36:01.483913 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:01.483797 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fdhrx" podStartSLOduration=1.964028618 podStartE2EDuration="11.48378798s" totalImagesPullingTime="9.519759362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:01.483060306 +0000 UTC m=+12.918561554" watchObservedRunningTime="2026-10-30 18:36:01.48378798 +0000 UTC m=+12.919289234" Oct 30 18:36:02.137221 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:02.137131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:02.137221 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:02.137171 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:02.137452 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:02.137138 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:02.137452 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:02.137271 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:02.137452 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:02.137373 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:02.137580 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:02.137480 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:02.477240 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:02.477162 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" event={"ID":"f2b77242-e22d-4689-9fd6-545ef1da88c4","Type":"ContainerStarted","Data":"2bd650b6a50202dd83e0533c20e168cf94f1327a64381f229bad3303a8339075"} Oct 30 18:36:02.480000 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:02.479965 2573 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="3dc775cd7937ff2a70d32dd784a30c4cf8657045518262deec8f99a742540594" exitCode=0 Oct 30 18:36:02.480172 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:02.480050 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snt75" event={"ID":"89a132e7-7020-4c9e-accd-89b0a61d73b0","Type":"ContainerDied","Data":"3dc775cd7937ff2a70d32dd784a30c4cf8657045518262deec8f99a742540594"} Oct 30 18:36:02.506735 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:02.506577 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" podStartSLOduration=2.806912051 podStartE2EDuration="12.506560504s" totalImagesPullingTime="9.699648453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:01.501929304 +0000 UTC m=+12.937430552" watchObservedRunningTime="2026-10-30 18:36:02.506560504 +0000 UTC m=+13.942061781" Oct 30 18:36:02.507187 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:02.507161 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" podStartSLOduration=0.562044817 podStartE2EDuration="12.507151207s" totalImagesPullingTime="11.94510639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:02.506510131 +0000 UTC m=+13.942011380" watchObservedRunningTime="2026-10-30 18:36:02.507151207 +0000 UTC m=+13.942652445" Oct 30 18:36:04.136162 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:04.136128 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:04.136886 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:04.136172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:04.136886 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:04.136268 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:04.136886 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:04.136345 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:04.136886 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:04.136358 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:04.136886 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:04.136440 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:04.487544 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:04.487425 2573 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="e677eee65bd01e2520544795b1f03d4eb516e7426986cb09faf89413103ad4e9" exitCode=0 Oct 30 18:36:04.487544 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:04.487488 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snt75" event={"ID":"89a132e7-7020-4c9e-accd-89b0a61d73b0","Type":"ContainerDied","Data":"e677eee65bd01e2520544795b1f03d4eb516e7426986cb09faf89413103ad4e9"} Oct 30 18:36:05.738763 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:05.738722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:05.739415 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:05.738872 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:36:05.739415 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:05.738952 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs podName:871c9271-d966-4c0d-b367-be51d1d4a79d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:21.738928981 +0000 UTC m=+33.174430226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs") pod "network-metrics-daemon-qj49w" (UID: "871c9271-d966-4c0d-b367-be51d1d4a79d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:36:05.839670 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:05.839600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:05.839873 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:05.839696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:05.839873 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:05.839791 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:36:05.839873 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:05.839814 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:36:05.839873 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:05.839817 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:36:05.839873 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:05.839837 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bv2pn for pod openshift-network-diagnostics/network-check-target-cbh4k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:36:05.840062 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:05.839882 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret podName:7d77554d-8094-4546-8fe7-0edf66752a5e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:21.839862074 +0000 UTC m=+33.275363301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret") pod "global-pull-secret-syncer-628cx" (UID: "7d77554d-8094-4546-8fe7-0edf66752a5e") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:36:05.840062 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:05.839900 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn podName:b52d343a-4ee8-474d-b2ab-6fc96edde2fd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:21.839894157 +0000 UTC m=+33.275395382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bv2pn" (UniqueName: "kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn") pod "network-check-target-cbh4k" (UID: "b52d343a-4ee8-474d-b2ab-6fc96edde2fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:36:06.137139 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:06.137061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:06.137325 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:06.137059 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:06.137325 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:06.137191 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:06.137325 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:06.137307 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:06.137438 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:06.137076 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:06.137438 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:06.137419 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:07.500542 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:07.500284 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"586b1d9b6ea8ff93ebc0979b9117a2f13216ba0bf3dedf7d4af34e2387a8997a"} Oct 30 18:36:07.501094 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:07.500576 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"a7ec27cf5874295c0308bf0ac64dbbc3df1d02de4b1c21eecffe052493eb252b"} Oct 30 18:36:07.501094 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:07.500627 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"328f5df66ea08bee704d306a2038ce059778c73c505ac512b3132a04db3e5a76"} Oct 30 18:36:07.501094 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:07.500655 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"5339f895ad06ab02ee59332b00ec6026855f9b8d4857b846acd9ceea6f3f3ae4"} Oct 30 18:36:07.501094 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:07.500681 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"f3e6ddb7a4f1041fce85c159756933737f4f244348034bac430924ce09afecfc"} Oct 30 18:36:07.501094 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:07.500707 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"7552267fce9101b53c7a90fd524942c0d85f818d4ddeda6e45b80e7f9c3ff43c"} Oct 30 18:36:07.503012 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:07.502987 2573 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="8f60dbcfc8429339cbf23e2cbdb66db36d5c0ed6a8e80d5503e0eebee4e3e537" exitCode=0 Oct 30 18:36:07.503113 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:07.503040 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snt75" event={"ID":"89a132e7-7020-4c9e-accd-89b0a61d73b0","Type":"ContainerDied","Data":"8f60dbcfc8429339cbf23e2cbdb66db36d5c0ed6a8e80d5503e0eebee4e3e537"} Oct 30 18:36:08.136720 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:08.136485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:08.136907 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:08.136506 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:08.136907 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:08.136798 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:08.136907 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:08.136546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:08.136907 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:08.136884 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:08.137029 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:08.136962 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:08.508362 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:08.508279 2573 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="384a47bca7b28e6fcb71a2460e5fb84005c9171bedc6093b6254d7a4051d7fa5" exitCode=0 Oct 30 18:36:08.508362 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:08.508334 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snt75" event={"ID":"89a132e7-7020-4c9e-accd-89b0a61d73b0","Type":"ContainerDied","Data":"384a47bca7b28e6fcb71a2460e5fb84005c9171bedc6093b6254d7a4051d7fa5"} Oct 30 18:36:09.514091 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:09.514045 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"b57b2f5a2986986ef602114a15adfd27baa8d4f823937b9c29e43a7b663f9364"} Oct 30 18:36:09.517321 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:09.517292 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snt75" event={"ID":"89a132e7-7020-4c9e-accd-89b0a61d73b0","Type":"ContainerStarted","Data":"dbbec08ca8364ec210da097918e9533949336e2dc06dd09cffcbaea0bfef4bcf"} Oct 30 18:36:09.542249 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:09.542197 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-snt75" podStartSLOduration=7.008134843 podStartE2EDuration="19.542183841s" totalImagesPullingTime="12.534048998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:09.540702975 +0000 UTC m=+20.976204223" watchObservedRunningTime="2026-10-30 18:36:09.542183841 +0000 UTC m=+20.977685090" Oct 30 18:36:10.136333 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:10.136299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:10.136516 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:10.136299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:10.136516 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:10.136425 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:10.136516 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:10.136476 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:10.136725 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:10.136544 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:10.136725 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:10.136617 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:11.526324 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:11.526285 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" event={"ID":"a21962c6-ef9b-413f-abe8-a93835acebbc","Type":"ContainerStarted","Data":"1efdd073c1ec6128677449329d4ad805acfbc94446b050fab27d47fc61935053"} Oct 30 18:36:11.527040 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:11.526595 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:36:11.527040 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:11.526706 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:36:11.544104 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:11.544028 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:36:11.544249 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:11.544230 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:36:11.555791 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:11.555733 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" podStartSLOduration=5.943652451 podStartE2EDuration="22.555718569s" totalImagesPullingTime="16.612066118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:11.554028734 +0000 UTC m=+22.989529983" watchObservedRunningTime="2026-10-30 18:36:11.555718569 +0000 UTC m=+22.991219835" Oct 30 18:36:12.136847 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:12.136808 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:12.136847 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:12.136849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:12.137045 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:12.136949 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:12.137045 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:12.137039 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:12.137116 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:12.137036 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:12.137178 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:12.137110 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:12.528350 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:12.528270 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:36:13.306880 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:13.306573 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cbh4k"] Oct 30 18:36:13.307095 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:13.306900 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:13.307095 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:13.307062 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:13.307229 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:13.307100 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-628cx"] Oct 30 18:36:13.307229 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:13.307192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:13.307312 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:13.307264 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:13.310322 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:13.310298 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qj49w"] Oct 30 18:36:13.310450 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:13.310404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:13.310525 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:13.310508 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:13.367973 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:13.367931 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:36:15.136662 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:15.136623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:15.137072 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:15.136623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:15.137072 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:15.136739 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:15.137072 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:15.136639 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:15.137072 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:15.136923 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:15.137072 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:15.136817 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:17.131671 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:17.131638 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:36:17.132209 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:17.131806 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:36:17.132325 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:17.132310 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:36:17.136232 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:17.136216 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:17.136311 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:17.136240 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:17.136370 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:17.136331 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:17.136421 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:17.136396 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:17.136461 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:17.136443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:17.136583 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:17.136562 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:19.136915 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:19.136880 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:19.137378 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:19.136983 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:19.137378 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:19.136997 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:19.137378 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:19.137008 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:19.137378 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:19.137077 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:19.137378 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:19.137130 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:21.136390 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:21.136357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:21.136926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:21.136357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:21.136926 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.136464 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:21.136926 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.136538 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:21.136926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:21.136370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:21.136926 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.136647 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:21.750857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:21.750821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:21.751035 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.750946 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:36:21.751035 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.751005 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs podName:871c9271-d966-4c0d-b367-be51d1d4a79d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:53.750987915 +0000 UTC m=+65.186489153 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs") pod "network-metrics-daemon-qj49w" (UID: "871c9271-d966-4c0d-b367-be51d1d4a79d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:36:21.851388 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:21.851344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:21.851599 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:21.851433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:21.851599 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.851504 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:36:21.851599 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.851553 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:36:21.851599 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.851569 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:36:21.851599 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.851582 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bv2pn for pod openshift-network-diagnostics/network-check-target-cbh4k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:36:21.851832 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.851570 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret podName:7d77554d-8094-4546-8fe7-0edf66752a5e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:53.851555529 +0000 UTC m=+65.287056755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret") pod "global-pull-secret-syncer-628cx" (UID: "7d77554d-8094-4546-8fe7-0edf66752a5e") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:36:21.851832 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:21.851661 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn podName:b52d343a-4ee8-474d-b2ab-6fc96edde2fd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:53.851646839 +0000 UTC m=+65.287148070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bv2pn" (UniqueName: "kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn") pod "network-check-target-cbh4k" (UID: "b52d343a-4ee8-474d-b2ab-6fc96edde2fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:36:22.429129 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:22.428926 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-msdfn_c8994e33-9b53-41d7-8fa8-04463a3ae647/dns-node-resolver/0.log" Oct 30 18:36:23.137027 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:23.136986 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:23.137216 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:23.137006 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:23.137216 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:23.137118 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:23.137216 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:23.137146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:23.137371 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:23.137287 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:23.137421 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:23.137385 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:23.412750 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:23.412678 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-slltd_272dcf18-8186-45d9-9316-8c5a0efc951b/node-ca/0.log" Oct 30 18:36:25.136500 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:25.136461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:25.136934 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:25.136473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:25.136934 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:25.136584 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:25.136934 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:25.136672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:25.136934 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:25.136473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:25.136934 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:25.136747 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:27.136719 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:27.136671 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:27.137127 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:27.136737 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:27.137127 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:27.136744 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:27.137127 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:27.136837 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:27.137127 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:27.136949 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:27.137127 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:27.137058 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:29.136965 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:29.136925 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:29.137401 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:29.137027 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:29.137401 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:29.137110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:29.137401 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:29.137210 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:29.137401 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:29.137272 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:29.137401 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:29.137391 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:31.136998 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:31.136961 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:31.137462 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:31.137021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:31.137462 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:31.137026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:31.137462 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:31.137126 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:31.137462 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:31.137243 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:31.137462 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:31.137342 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:33.136863 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:33.136830 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:33.137353 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:33.136834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:33.137353 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:33.136959 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qj49w" podUID="871c9271-d966-4c0d-b367-be51d1d4a79d" Oct 30 18:36:33.137353 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:33.136845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:33.137353 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:33.137025 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbh4k" podUID="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" Oct 30 18:36:33.137353 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:33.137090 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-628cx" podUID="7d77554d-8094-4546-8fe7-0edf66752a5e" Oct 30 18:36:34.925950 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.925865 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeReady" Oct 30 18:36:34.926392 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.926001 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 18:36:34.974789 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.974749 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f92sm"] Oct 30 18:36:34.977775 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.977748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:36:34.978304 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.978285 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-45mzz"] Oct 30 18:36:34.981170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.981155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:34.984821 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.984794 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:36:34.984961 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.984847 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:36:34.985022 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.984994 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5rxk7\"" Oct 30 18:36:34.987592 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.987571 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:36:34.987758 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.987731 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qs4k8\"" Oct 30 18:36:34.987839 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.987798 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:36:34.988174 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.988156 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:36:34.988907 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.988886 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f92sm"] Oct 30 18:36:34.991786 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.991765 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45mzz"] Oct 30 18:36:34.994921 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.994898 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jbqcr"] Oct 30 18:36:34.999543 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:34.999523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.011532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.011505 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jbqcr"] Oct 30 18:36:35.044663 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.044599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hs8\" (UniqueName: \"kubernetes.io/projected/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-kube-api-access-q8hs8\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:36:35.044663 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.044659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-cert\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:36:35.085886 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.085846 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:36:35.085886 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.085846 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:36:35.086100 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.085919 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:36:35.086100 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.085942 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fb5r\"" Oct 30 18:36:35.086100 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.085973 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:36:35.136621 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.136587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:35.136744 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.136658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:35.136744 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.136660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86b7d9-2e05-4f95-b002-931249ab66e2-config-volume\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hs8\" (UniqueName: \"kubernetes.io/projected/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-kube-api-access-q8hs8\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d52x\" (UniqueName: \"kubernetes.io/projected/8d86b7d9-2e05-4f95-b002-931249ab66e2-kube-api-access-8d52x\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-cert\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d86b7d9-2e05-4f95-b002-931249ab66e2-metrics-tls\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d86b7d9-2e05-4f95-b002-931249ab66e2-tmp-dir\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-data-volume\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqjj\" (UniqueName: \"kubernetes.io/projected/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-api-access-bcqjj\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.146749 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.145869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-crio-socket\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.151785 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.151761 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-cert\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:36:35.154845 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.154817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hs8\" (UniqueName: \"kubernetes.io/projected/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-kube-api-access-q8hs8\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:36:35.186632 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.186521 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:36:35.186632 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.186521 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-phzbf\"" Oct 30 18:36:35.186632 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.186521 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgk4h\"" Oct 30 18:36:35.186632 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.186529 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:36:35.186632 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.186583 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:36:35.187754 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.187739 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:36:35.187794 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.187768 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnnzq\"" Oct 30 18:36:35.247124 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.247249 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247176 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86b7d9-2e05-4f95-b002-931249ab66e2-config-volume\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.247249 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d52x\" (UniqueName: \"kubernetes.io/projected/8d86b7d9-2e05-4f95-b002-931249ab66e2-kube-api-access-8d52x\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.247249 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d86b7d9-2e05-4f95-b002-931249ab66e2-metrics-tls\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.247495 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d86b7d9-2e05-4f95-b002-931249ab66e2-tmp-dir\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.247679 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-data-volume\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.247679 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqjj\" (UniqueName: \"kubernetes.io/projected/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-api-access-bcqjj\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.247679 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.247833 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-crio-socket\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.247886 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247853 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-data-volume\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.248039 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247902 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d86b7d9-2e05-4f95-b002-931249ab66e2-tmp-dir\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.248119 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.247925 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-crio-socket\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.248180 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.248135 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86b7d9-2e05-4f95-b002-931249ab66e2-config-volume\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.248252 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.248237 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.249728 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.249707 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d86b7d9-2e05-4f95-b002-931249ab66e2-metrics-tls\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.249857 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.249840 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.255943 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.255920 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d52x\" (UniqueName: \"kubernetes.io/projected/8d86b7d9-2e05-4f95-b002-931249ab66e2-kube-api-access-8d52x\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.256029 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.255962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqjj\" (UniqueName: \"kubernetes.io/projected/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-api-access-bcqjj\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.291065 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.291030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:36:35.301361 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.301335 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:35.315314 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.315281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:36:35.585078 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.584986 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45mzz"] Oct 30 18:36:35.588172 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.588144 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f92sm"] Oct 30 18:36:35.589479 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:35.589359 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d86b7d9_2e05_4f95_b002_931249ab66e2.slice/crio-7b42b15eb9b8d77cc4bcdc91e1f36a8c3e86bede2d68f98a109038d20725dd9c WatchSource:0}: Error finding container 7b42b15eb9b8d77cc4bcdc91e1f36a8c3e86bede2d68f98a109038d20725dd9c: Status 404 returned error can't find the container with id 7b42b15eb9b8d77cc4bcdc91e1f36a8c3e86bede2d68f98a109038d20725dd9c Oct 30 18:36:35.592110 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:35.592085 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbbc1ad_2bb2_469a_8b29_94760bce83cc.slice/crio-bfb9e46b67dc8a2b747d12243c10961fa20445e870fcb4da4157f88ccfe476cf WatchSource:0}: Error finding container bfb9e46b67dc8a2b747d12243c10961fa20445e870fcb4da4157f88ccfe476cf: Status 404 returned error can't find the container with id bfb9e46b67dc8a2b747d12243c10961fa20445e870fcb4da4157f88ccfe476cf Oct 30 18:36:35.614705 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:35.614677 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jbqcr"] Oct 30 18:36:35.620094 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:35.620060 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353c68dc_e91f_445b_b4f0_386cf1a9c8e7.slice/crio-433f3b2041f96c84031e68c7d26bdf986bda69de52d983aaa7b3f97bc2f6f05c WatchSource:0}: Error finding container 433f3b2041f96c84031e68c7d26bdf986bda69de52d983aaa7b3f97bc2f6f05c: Status 404 returned error can't find the container with id 433f3b2041f96c84031e68c7d26bdf986bda69de52d983aaa7b3f97bc2f6f05c Oct 30 18:36:36.586757 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.586715 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jbqcr" event={"ID":"353c68dc-e91f-445b-b4f0-386cf1a9c8e7","Type":"ContainerStarted","Data":"a78e3667f032e2aab17c9f7ca04eee9049cbd6cbc7e7f4ed195b48915c4b4582"} Oct 30 18:36:36.587215 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.586778 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jbqcr" event={"ID":"353c68dc-e91f-445b-b4f0-386cf1a9c8e7","Type":"ContainerStarted","Data":"4ce14a37993c09f2059f19d02859bea07e8f4f2a03a540a96dbff709d0df52f9"} Oct 30 18:36:36.587215 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.586808 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jbqcr" event={"ID":"353c68dc-e91f-445b-b4f0-386cf1a9c8e7","Type":"ContainerStarted","Data":"433f3b2041f96c84031e68c7d26bdf986bda69de52d983aaa7b3f97bc2f6f05c"} Oct 30 18:36:36.588451 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.588421 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45mzz" event={"ID":"8d86b7d9-2e05-4f95-b002-931249ab66e2","Type":"ContainerStarted","Data":"7b42b15eb9b8d77cc4bcdc91e1f36a8c3e86bede2d68f98a109038d20725dd9c"} Oct 30 18:36:36.589832 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.589801 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f92sm" event={"ID":"fbbbc1ad-2bb2-469a-8b29-94760bce83cc","Type":"ContainerStarted","Data":"bfb9e46b67dc8a2b747d12243c10961fa20445e870fcb4da4157f88ccfe476cf"} Oct 30 18:36:36.857576 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.857540 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4"] Oct 30 18:36:36.862143 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.860842 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:36.867044 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.867010 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 18:36:36.867044 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.867030 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:36:36.867044 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.867044 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:36:36.867319 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.867084 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 18:36:36.867319 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.867196 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:36:36.867319 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.867270 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-k7fv4\"" Oct 30 18:36:36.872883 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.872853 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4"] Oct 30 18:36:36.962959 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.962875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:36.962959 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.962932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:36.963316 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.962998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:36.963316 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:36.963081 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtcjs\" (UniqueName: \"kubernetes.io/projected/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-kube-api-access-qtcjs\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.064369 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.064321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.064583 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.064423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.064583 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.064502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcjs\" (UniqueName: \"kubernetes.io/projected/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-kube-api-access-qtcjs\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.064583 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.064550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.065201 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.065175 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.067758 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.067718 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.070556 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.070528 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.075275 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.075226 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcjs\" (UniqueName: \"kubernetes.io/projected/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-kube-api-access-qtcjs\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.178763 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.178721 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:36:37.463350 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.463231 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4"] Oct 30 18:36:37.466938 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:37.466906 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ef9c9a_e45e_447f_b68c_56ef8d59ab96.slice/crio-7acf902a75afca9f4419ca945c7c017a5fe82b3cefb108cab39234025d4c4f25 WatchSource:0}: Error finding container 7acf902a75afca9f4419ca945c7c017a5fe82b3cefb108cab39234025d4c4f25: Status 404 returned error can't find the container with id 7acf902a75afca9f4419ca945c7c017a5fe82b3cefb108cab39234025d4c4f25 Oct 30 18:36:37.594193 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:37.594155 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" event={"ID":"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96","Type":"ContainerStarted","Data":"7acf902a75afca9f4419ca945c7c017a5fe82b3cefb108cab39234025d4c4f25"} Oct 30 18:36:38.600537 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:38.599891 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jbqcr" event={"ID":"353c68dc-e91f-445b-b4f0-386cf1a9c8e7","Type":"ContainerStarted","Data":"422f55b21b95038570944a3657376b05db1b9088d9da4980c608a149934b150f"} Oct 30 18:36:38.602085 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:38.602057 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45mzz" event={"ID":"8d86b7d9-2e05-4f95-b002-931249ab66e2","Type":"ContainerStarted","Data":"8628dc583a7a411f80d9c26612a06bb0fa6d5ceaefabe6df8d45d06688608573"} Oct 30 18:36:38.602219 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:38.602113 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45mzz" event={"ID":"8d86b7d9-2e05-4f95-b002-931249ab66e2","Type":"ContainerStarted","Data":"714b9f7be4fa8330d2722626766aa489d2c02469966ff12484db78bc967f0154"} Oct 30 18:36:38.602219 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:38.602208 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:38.623317 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:38.623262 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jbqcr" podStartSLOduration=2.258590614 podStartE2EDuration="4.623243777s" totalImagesPullingTime="2.364653163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:38.622851944 +0000 UTC m=+50.058353193" watchObservedRunningTime="2026-10-30 18:36:38.623243777 +0000 UTC m=+50.058745023" Oct 30 18:36:38.649086 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:38.649028 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-45mzz" podStartSLOduration=2.726486014 podStartE2EDuration="4.649013662s" totalImagesPullingTime="1.922527648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:38.64833645 +0000 UTC m=+50.083837698" watchObservedRunningTime="2026-10-30 18:36:38.649013662 +0000 UTC m=+50.084514912" Oct 30 18:36:39.607194 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:39.607157 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" event={"ID":"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96","Type":"ContainerStarted","Data":"4a82856ea4c6a2bb27b4854854ecfdef349f29f563386958e0ad134e9b835934"} Oct 30 18:36:39.607668 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:39.607240 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" event={"ID":"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96","Type":"ContainerStarted","Data":"126371a0d074c056b0ea83678aa66a5d772acbbefbd33aa358b3ea70a850a8a5"} Oct 30 18:36:39.608722 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:39.608698 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f92sm" event={"ID":"fbbbc1ad-2bb2-469a-8b29-94760bce83cc","Type":"ContainerStarted","Data":"1d8594a2f04819704acfd24d19998432cbfedd16ab833b69385d27c88c857bde"} Oct 30 18:36:39.625546 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:39.625491 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" podStartSLOduration=2.169124162 podStartE2EDuration="3.625476937s" totalImagesPullingTime="1.456352775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:39.624672713 +0000 UTC m=+51.060173962" watchObservedRunningTime="2026-10-30 18:36:39.625476937 +0000 UTC m=+51.060978184" Oct 30 18:36:39.643017 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:39.642959 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f92sm" podStartSLOduration=2.68832983 podStartE2EDuration="5.642942872s" totalImagesPullingTime="2.954613042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:39.641811067 +0000 UTC m=+51.077312314" watchObservedRunningTime="2026-10-30 18:36:39.642942872 +0000 UTC m=+51.078444119" Oct 30 18:36:41.215675 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.215638 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s"] Oct 30 18:36:41.218838 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.218819 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rrqjz"] Oct 30 18:36:41.218957 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.218933 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.221945 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.221925 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.224030 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.224016 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 18:36:41.224087 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.224050 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cxpnh\"" Oct 30 18:36:41.255285 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.255252 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:36:41.261222 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.261196 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:36:41.261398 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.261384 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:36:41.261623 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.261580 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dgrqn\"" Oct 30 18:36:41.264539 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.264515 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s"] Oct 30 18:36:41.270422 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.270404 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:36:41.274698 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.274668 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg"] Oct 30 18:36:41.279001 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.278972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.292231 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.292206 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg"] Oct 30 18:36:41.323532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.323500 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:36:41.323780 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.323764 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 18:36:41.323847 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.323837 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t5wt5\"" Oct 30 18:36:41.355198 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.355163 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 18:36:41.401401 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401369 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.401513 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401404 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-wtmp\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.401513 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-tls\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.401513 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.401692 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmp8\" (UniqueName: \"kubernetes.io/projected/73bfd18d-99d5-4967-b748-00d94f5dc398-kube-api-access-zqmp8\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.401692 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.401778 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.401827 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6dx\" (UniqueName: \"kubernetes.io/projected/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-kube-api-access-hn6dx\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.401827 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-textfile\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.401929 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401862 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-root\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.401977 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d8e52eca-6d99-4f5c-bbee-9f9559130add-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.402032 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.401984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.402032 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.402010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-sys\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.402110 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.402040 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.402143 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.402134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-metrics-client-ca\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.402173 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.402155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.402275 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.402212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.402275 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.402237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.402275 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.402266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2b6n\" (UniqueName: \"kubernetes.io/projected/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-api-access-v2b6n\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.502926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.502844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-metrics-client-ca\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.502926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.502883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.502926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.502906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.503160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.502941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.503160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.502965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b6n\" (UniqueName: \"kubernetes.io/projected/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-api-access-v2b6n\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.503160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.502983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.503160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-wtmp\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-tls\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmp8\" (UniqueName: \"kubernetes.io/projected/73bfd18d-99d5-4967-b748-00d94f5dc398-kube-api-access-zqmp8\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503160 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6dx\" (UniqueName: \"kubernetes.io/projected/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-kube-api-access-hn6dx\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503204 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-wtmp\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-textfile\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-root\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d8e52eca-6d99-4f5c-bbee-9f9559130add-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-sys\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.503514 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503482 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-textfile\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.504005 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503568 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-metrics-client-ca\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.504005 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503694 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.504005 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503745 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d8e52eca-6d99-4f5c-bbee-9f9559130add-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.504005 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-sys\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.504211 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.504078 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.504316 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.504293 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.504379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.503393 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-root\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.504379 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.504326 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.506009 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.505980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.506232 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.506207 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.506547 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.506527 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.506829 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.506809 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-tls\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.506928 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.506905 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.507191 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.507172 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.511667 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.511641 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b6n\" (UniqueName: \"kubernetes.io/projected/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-api-access-v2b6n\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.511755 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.511694 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmp8\" (UniqueName: \"kubernetes.io/projected/73bfd18d-99d5-4967-b748-00d94f5dc398-kube-api-access-zqmp8\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.512061 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.512045 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6dx\" (UniqueName: \"kubernetes.io/projected/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-kube-api-access-hn6dx\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.536645 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.536593 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:36:41.543257 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.543220 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:36:41.556415 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:41.556388 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bfd18d_99d5_4967_b748_00d94f5dc398.slice/crio-4020703a95484315726b9502ccaafd08d26dc3e0287b28771549d24120e66621 WatchSource:0}: Error finding container 4020703a95484315726b9502ccaafd08d26dc3e0287b28771549d24120e66621: Status 404 returned error can't find the container with id 4020703a95484315726b9502ccaafd08d26dc3e0287b28771549d24120e66621 Oct 30 18:36:41.595300 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.595267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:36:41.617403 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.617360 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrqjz" event={"ID":"73bfd18d-99d5-4967-b748-00d94f5dc398","Type":"ContainerStarted","Data":"4020703a95484315726b9502ccaafd08d26dc3e0287b28771549d24120e66621"} Oct 30 18:36:41.824665 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.824565 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s"] Oct 30 18:36:41.827144 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:41.827108 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb2bc4b_5c95_469f_bdca_5fe3862f23ad.slice/crio-5c4196f8503be0d4a88dda1f4100de07bf9130710453e0c72bbd866f520390b0 WatchSource:0}: Error finding container 5c4196f8503be0d4a88dda1f4100de07bf9130710453e0c72bbd866f520390b0: Status 404 returned error can't find the container with id 5c4196f8503be0d4a88dda1f4100de07bf9130710453e0c72bbd866f520390b0 Oct 30 18:36:41.872574 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:41.872555 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg"] Oct 30 18:36:41.874625 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:41.874580 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e52eca_6d99_4f5c_bbee_9f9559130add.slice/crio-7a2c398fa990b15181a4e101f345ddf18c076806db928e3f9d01c1dbbb37d7eb WatchSource:0}: Error finding container 7a2c398fa990b15181a4e101f345ddf18c076806db928e3f9d01c1dbbb37d7eb: Status 404 returned error can't find the container with id 7a2c398fa990b15181a4e101f345ddf18c076806db928e3f9d01c1dbbb37d7eb Oct 30 18:36:42.304815 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.304778 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:36:42.310178 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.308671 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.313872 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.313663 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:36:42.313872 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.313726 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:36:42.313872 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.313754 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:36:42.313872 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.313807 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:36:42.313872 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.313844 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:36:42.314869 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.314844 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:36:42.314990 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.314894 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:36:42.314990 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.314950 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:36:42.315094 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.315023 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vtmwt\"" Oct 30 18:36:42.315094 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.315043 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:36:42.322576 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.322548 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:36:42.410532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410364 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410701 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.410926 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.410809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.511962 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.511922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512102 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.511977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512102 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512102 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512102 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512285 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512285 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512285 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512285 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512285 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512521 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512521 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.512521 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.512356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.515180 ip-10-0-133-96 kubenswrapper[2573]: E1030 18:36:42.513387 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle podName:4d6ce326-8d12-49b8-90cd-cf42ab792490 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.013364929 +0000 UTC m=+54.448866171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490") : configmap references non-existent config key: ca-bundle.crt Oct 30 18:36:42.515180 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.513942 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.515180 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.514503 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.523709 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.523677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.527573 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.527550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.529153 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.529133 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.536511 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.536483 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.536692 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.536524 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.536990 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.536974 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.538209 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.538177 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.542048 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.542025 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.542280 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.542260 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.544125 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.544099 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:42.625827 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.625717 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrqjz" event={"ID":"73bfd18d-99d5-4967-b748-00d94f5dc398","Type":"ContainerStarted","Data":"36d8d3c07432c46a3e13ca012defd3b9b69d9dff4dde4bf6c7c26cac29d40e31"} Oct 30 18:36:42.627666 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.627634 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" event={"ID":"d8e52eca-6d99-4f5c-bbee-9f9559130add","Type":"ContainerStarted","Data":"7a2c398fa990b15181a4e101f345ddf18c076806db928e3f9d01c1dbbb37d7eb"} Oct 30 18:36:42.631544 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.631493 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" event={"ID":"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad","Type":"ContainerStarted","Data":"244c0eec2bda9842c69210e52ead9897e3ab972bf4ba9b49210880e5d126c7d6"} Oct 30 18:36:42.631544 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.631543 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" event={"ID":"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad","Type":"ContainerStarted","Data":"30c4daa15d7e2073a9be2db536812e5df71b1c7fafb0156aed3298d1ee42acd7"} Oct 30 18:36:42.631544 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:42.631560 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" event={"ID":"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad","Type":"ContainerStarted","Data":"5c4196f8503be0d4a88dda1f4100de07bf9130710453e0c72bbd866f520390b0"} Oct 30 18:36:43.017574 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.017538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:43.018282 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.018252 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:43.057830 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.057794 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5f79687c-qhrj9"] Oct 30 18:36:43.062572 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.062544 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.071257 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.071233 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 18:36:43.072220 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.072198 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 18:36:43.072363 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.072225 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 18:36:43.072363 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.072209 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 18:36:43.072363 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.072290 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 18:36:43.072363 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.072290 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 18:36:43.072598 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.072406 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-slnc4\"" Oct 30 18:36:43.072598 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.072456 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 18:36:43.075618 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.075580 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5f79687c-qhrj9"] Oct 30 18:36:43.219471 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.219448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.219561 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.219483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.219561 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.219505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.219648 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.219564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.219648 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.219642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.219712 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.219666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5x6c\" (UniqueName: \"kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.219712 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.219694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.232854 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.232832 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7fd85567c4-mkwgs"] Oct 30 18:36:43.237687 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.237668 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.240240 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.240220 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:36:43.252825 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.252797 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fd85567c4-mkwgs"] Oct 30 18:36:43.271922 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.271903 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 18:36:43.272020 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.272001 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 18:36:43.272096 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.272068 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 18:36:43.272170 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.272159 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4ebv7a0m728dl\"" Oct 30 18:36:43.272411 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.272397 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4z4br\"" Oct 30 18:36:43.272465 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.272446 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 18:36:43.272533 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.272517 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 18:36:43.320778 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.320753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.321090 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.320823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.321090 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.320852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.321090 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.320874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.321090 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.320899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.321090 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.320914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.321090 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.320938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5x6c\" (UniqueName: \"kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.322347 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.321865 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.322347 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.322269 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.322347 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.322268 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.322823 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.322802 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.324407 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.324391 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.324556 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.324538 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.331999 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.331972 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5x6c\" (UniqueName: \"kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.378644 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.377689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-metrics-client-ca\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xb5h\" (UniqueName: \"kubernetes.io/projected/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-kube-api-access-5xb5h\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422173 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-tmp\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-grpc-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.422726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.422408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523327 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523327 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-metrics-client-ca\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523529 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xb5h\" (UniqueName: \"kubernetes.io/projected/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-kube-api-access-5xb5h\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523529 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523374 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523529 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-tmp\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523529 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-grpc-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523847 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523847 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.523847 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.523809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.524529 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.524486 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-tmp\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.524529 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.524495 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-metrics-client-ca\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.527027 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.527004 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.527027 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.527021 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.527174 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.527004 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-grpc-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.527174 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.527083 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.527349 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.527331 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.527589 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.527555 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.531855 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.531829 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:36:43.533956 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.533936 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xb5h\" (UniqueName: \"kubernetes.io/projected/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-kube-api-access-5xb5h\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.535918 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:43.535892 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6ce326_8d12_49b8_90cd_cf42ab792490.slice/crio-d43a365bed980790e8a1fa057ee1e434d810db1209c724fd4f17c7c67e9da874 WatchSource:0}: Error finding container d43a365bed980790e8a1fa057ee1e434d810db1209c724fd4f17c7c67e9da874: Status 404 returned error can't find the container with id d43a365bed980790e8a1fa057ee1e434d810db1209c724fd4f17c7c67e9da874 Oct 30 18:36:43.569384 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.569354 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:43.644211 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.638733 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" event={"ID":"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad","Type":"ContainerStarted","Data":"2a6d27631bb6d71bba586233af4e057d5f1be19c38a91f98b18e860b326f928a"} Oct 30 18:36:43.644211 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.644198 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerStarted","Data":"d43a365bed980790e8a1fa057ee1e434d810db1209c724fd4f17c7c67e9da874"} Oct 30 18:36:43.647911 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.647668 2573 generic.go:356] "Generic (PLEG): container finished" podID="73bfd18d-99d5-4967-b748-00d94f5dc398" containerID="36d8d3c07432c46a3e13ca012defd3b9b69d9dff4dde4bf6c7c26cac29d40e31" exitCode=0 Oct 30 18:36:43.647911 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.647786 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrqjz" event={"ID":"73bfd18d-99d5-4967-b748-00d94f5dc398","Type":"ContainerDied","Data":"36d8d3c07432c46a3e13ca012defd3b9b69d9dff4dde4bf6c7c26cac29d40e31"} Oct 30 18:36:43.650561 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.650528 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" event={"ID":"d8e52eca-6d99-4f5c-bbee-9f9559130add","Type":"ContainerStarted","Data":"cff681f09241785bae4e1fe03dcd9eacc2a5f5072dfaa698ceadbef5baad0627"} Oct 30 18:36:43.650769 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.650591 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" event={"ID":"d8e52eca-6d99-4f5c-bbee-9f9559130add","Type":"ContainerStarted","Data":"49e8b88d63ab430c70f2d94925f392e6eb2be126f1cc9a4c74c9eeb1ecdabe37"} Oct 30 18:36:43.650769 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.650636 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" event={"ID":"d8e52eca-6d99-4f5c-bbee-9f9559130add","Type":"ContainerStarted","Data":"28c382a7dd9b52045cf80878e2a3d6a9f4771e76a57cb039521a60e33d000f74"} Oct 30 18:36:43.664251 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.664202 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" podStartSLOduration=1.577883935 podStartE2EDuration="2.664185906s" totalImagesPullingTime="1.086301971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:43.662883367 +0000 UTC m=+55.098384615" watchObservedRunningTime="2026-10-30 18:36:43.664185906 +0000 UTC m=+55.099687185" Oct 30 18:36:43.665460 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.665440 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5f79687c-qhrj9"] Oct 30 18:36:43.675678 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:43.675654 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd549b7_0627_4c6c_9f3e_e04f01607458.slice/crio-676128efd1ec3d1cabda721eefbfe19ede0f6625081a57a8ac54f1482ee85f68 WatchSource:0}: Error finding container 676128efd1ec3d1cabda721eefbfe19ede0f6625081a57a8ac54f1482ee85f68: Status 404 returned error can't find the container with id 676128efd1ec3d1cabda721eefbfe19ede0f6625081a57a8ac54f1482ee85f68 Oct 30 18:36:43.712859 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.711380 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" podStartSLOduration=1.385820094 podStartE2EDuration="2.711359398s" totalImagesPullingTime="1.325539304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:43.710864137 +0000 UTC m=+55.146365396" watchObservedRunningTime="2026-10-30 18:36:43.711359398 +0000 UTC m=+55.146860644" Oct 30 18:36:43.848011 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:43.847923 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fd85567c4-mkwgs"] Oct 30 18:36:43.851888 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:43.851859 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fc82ca_c9cf_43f1_871c_c156ea050ca6.slice/crio-bb627f0ff19dec1f579603dfb93d7d049ce7e150e4f7cde6802fa6d4ab735564 WatchSource:0}: Error finding container bb627f0ff19dec1f579603dfb93d7d049ce7e150e4f7cde6802fa6d4ab735564: Status 404 returned error can't find the container with id bb627f0ff19dec1f579603dfb93d7d049ce7e150e4f7cde6802fa6d4ab735564 Oct 30 18:36:44.548396 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:44.548366 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:36:44.657552 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:44.657513 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" event={"ID":"a5fc82ca-c9cf-43f1-871c-c156ea050ca6","Type":"ContainerStarted","Data":"bb627f0ff19dec1f579603dfb93d7d049ce7e150e4f7cde6802fa6d4ab735564"} Oct 30 18:36:44.661213 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:44.661180 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrqjz" event={"ID":"73bfd18d-99d5-4967-b748-00d94f5dc398","Type":"ContainerStarted","Data":"d51db7a28a203c24f9d53d0ea6793ccf08dfd5d3d65035df29486f6033d9d8ea"} Oct 30 18:36:44.661298 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:44.661252 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrqjz" event={"ID":"73bfd18d-99d5-4967-b748-00d94f5dc398","Type":"ContainerStarted","Data":"af07e86e241438d1ca649107288848c133003340ad866c999372d8199b5c5450"} Oct 30 18:36:44.668451 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:44.668385 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5f79687c-qhrj9" event={"ID":"8dd549b7-0627-4c6c-9f3e-e04f01607458","Type":"ContainerStarted","Data":"676128efd1ec3d1cabda721eefbfe19ede0f6625081a57a8ac54f1482ee85f68"} Oct 30 18:36:45.650206 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.647902 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rrqjz" podStartSLOduration=3.590449917 podStartE2EDuration="4.647880676s" totalImagesPullingTime="966.479642ms" totalInitContainerRuntime="90.951117ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:44.686086261 +0000 UTC m=+56.121587512" watchObservedRunningTime="2026-10-30 18:36:45.647880676 +0000 UTC m=+57.083381925" Oct 30 18:36:45.650206 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.648901 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c7bcfc979-g65nx"] Oct 30 18:36:45.653245 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.652559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.656508 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.656470 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 18:36:45.656670 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.656587 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:36:45.664481 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.664452 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c7bcfc979-g65nx"] Oct 30 18:36:45.676596 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.676560 2573 generic.go:356] "Generic (PLEG): container finished" podID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerID="8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96" exitCode=0 Oct 30 18:36:45.676780 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.676668 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerDied","Data":"8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96"} Oct 30 18:36:45.689825 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.689783 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 18:36:45.689825 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.689790 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 18:36:45.690523 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.690488 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gxldj\"" Oct 30 18:36:45.690663 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.690590 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-31139jp48a13i\"" Oct 30 18:36:45.848784 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.848741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.848957 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.848791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-audit-log\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.849339 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.849317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-tls\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.849397 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.849371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-metrics-server-audit-profiles\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.849445 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.849398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-client-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.849445 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.849419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-client-certs\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.849532 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.849482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbdq\" (UniqueName: \"kubernetes.io/projected/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-kube-api-access-wpbdq\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.950849 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.950820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.951003 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.950856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-audit-log\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.951003 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.950958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-tls\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.951003 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.950990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-metrics-server-audit-profiles\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.951189 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.951016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-client-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.951189 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.951049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-client-certs\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.951189 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.951083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbdq\" (UniqueName: \"kubernetes.io/projected/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-kube-api-access-wpbdq\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.951768 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.951505 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-audit-log\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.951768 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.951702 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.952105 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.952077 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-metrics-server-audit-profiles\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.954002 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.953972 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-client-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.954094 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.953987 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-tls\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.954411 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.954324 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-client-certs\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.965329 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.965293 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbdq\" (UniqueName: \"kubernetes.io/projected/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-kube-api-access-wpbdq\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:45.972338 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:45.972294 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:36:46.274066 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:46.274029 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c7bcfc979-g65nx"] Oct 30 18:36:46.278704 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:46.278665 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e4cb6c_9268_4dda_a3a8_c8bb503942fb.slice/crio-e0d1b0b56f4aa1b164fe8d1d0676e21f5355ddd50d625d3e9809d384e39bfab0 WatchSource:0}: Error finding container e0d1b0b56f4aa1b164fe8d1d0676e21f5355ddd50d625d3e9809d384e39bfab0: Status 404 returned error can't find the container with id e0d1b0b56f4aa1b164fe8d1d0676e21f5355ddd50d625d3e9809d384e39bfab0 Oct 30 18:36:46.682933 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:46.682833 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" event={"ID":"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb","Type":"ContainerStarted","Data":"e0d1b0b56f4aa1b164fe8d1d0676e21f5355ddd50d625d3e9809d384e39bfab0"} Oct 30 18:36:46.686058 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:46.685871 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" event={"ID":"a5fc82ca-c9cf-43f1-871c-c156ea050ca6","Type":"ContainerStarted","Data":"c960ebbb8b89cdec72d0f301f35cc2bfd7c7205a9fa3a82a915fc3dc0369265c"} Oct 30 18:36:46.686058 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:46.685953 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" event={"ID":"a5fc82ca-c9cf-43f1-871c-c156ea050ca6","Type":"ContainerStarted","Data":"a466c40ba19168d1c65ee391772010a7cdcde7d53fe1d41117da6da09564a852"} Oct 30 18:36:46.686058 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:46.685993 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" event={"ID":"a5fc82ca-c9cf-43f1-871c-c156ea050ca6","Type":"ContainerStarted","Data":"5903c3d3cab4fa2036310ae3bccd182ae010b771ee284f333c5d11f868f6ad17"} Oct 30 18:36:47.609042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.608974 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:47.618082 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.618039 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.634860 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.634834 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:36:47.637656 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.637631 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:36:47.638018 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.637980 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:36:47.643539 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.643514 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.651906 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8lvuvgi8ronlo\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652117 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652189 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652249 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652303 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kqrcb\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652326 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652393 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652426 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652456 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652481 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652528 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.652573 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:36:47.659661 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.656173 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670364 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670404 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670451 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lf6\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670542 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670639 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.670729 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.671386 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.670800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.674506 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.671053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.674506 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.672708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.674506 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.672766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.696034 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.695882 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerStarted","Data":"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c"} Oct 30 18:36:47.696034 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.695956 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerStarted","Data":"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284"} Oct 30 18:36:47.696034 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.695983 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerStarted","Data":"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656"} Oct 30 18:36:47.703438 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.703404 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" event={"ID":"a5fc82ca-c9cf-43f1-871c-c156ea050ca6","Type":"ContainerStarted","Data":"26d16d872d25ef9fe34709984a2e4067fd0dde14e7ad1eb8fa6a62521703b12b"} Oct 30 18:36:47.703548 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.703477 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" event={"ID":"a5fc82ca-c9cf-43f1-871c-c156ea050ca6","Type":"ContainerStarted","Data":"9b1aa1a5d2fcb7fc721b5088e3f80498d4108c5389a5e6d9d7f465ed12a43d06"} Oct 30 18:36:47.703548 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.703511 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" event={"ID":"a5fc82ca-c9cf-43f1-871c-c156ea050ca6","Type":"ContainerStarted","Data":"7897e579f3ec10052b4378582ed699793810fd31d2abeaaf4e6193e0529ff292"} Oct 30 18:36:47.704872 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.704794 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:47.710998 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.710240 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5f79687c-qhrj9" event={"ID":"8dd549b7-0627-4c6c-9f3e-e04f01607458","Type":"ContainerStarted","Data":"c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1"} Oct 30 18:36:47.747421 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.747314 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" podStartSLOduration=1.860041169 podStartE2EDuration="4.747290684s" totalImagesPullingTime="2.887249515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:47.746025756 +0000 UTC m=+59.181527004" watchObservedRunningTime="2026-10-30 18:36:47.747290684 +0000 UTC m=+59.182791934" Oct 30 18:36:47.771917 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.771842 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f5f79687c-qhrj9" podStartSLOduration=1.491859087 podStartE2EDuration="4.7718193s" totalImagesPullingTime="3.279960213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:47.76882785 +0000 UTC m=+59.204329100" watchObservedRunningTime="2026-10-30 18:36:47.7718193 +0000 UTC m=+59.207320549" Oct 30 18:36:47.773269 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773399 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773399 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773399 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773399 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lf6\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773399 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773703 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773703 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773703 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773703 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773703 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773933 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773933 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773933 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.773933 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.774124 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.774124 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.773979 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.774124 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.774072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.774124 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.774101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.774311 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.774126 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.774667 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.774550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.778444 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.776467 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.778444 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.776550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.778444 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.777094 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.778444 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.777725 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.781145 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.780396 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.781145 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.780967 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.781955 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.781929 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.782579 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.782548 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.782838 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.782788 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.784342 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.783075 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.784342 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.783682 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.784342 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.783735 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.784342 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.783755 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.784342 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.784134 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.784775 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.784644 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.785711 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.785662 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.786241 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.786200 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.788440 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.788395 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.789277 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.789205 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lf6\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:47.965038 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:47.964994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:48.267321 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.267257 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:48.269511 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:48.269481 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8565b888_5ed6_4d0f_a91c_87659bedfe00.slice/crio-71e42f68fd77314a41cf0580fb1c8ae2e24a30c25032800afd6bb05ad5da2786 WatchSource:0}: Error finding container 71e42f68fd77314a41cf0580fb1c8ae2e24a30c25032800afd6bb05ad5da2786: Status 404 returned error can't find the container with id 71e42f68fd77314a41cf0580fb1c8ae2e24a30c25032800afd6bb05ad5da2786 Oct 30 18:36:48.610997 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.610888 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-45mzz" Oct 30 18:36:48.715545 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.715500 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" event={"ID":"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb","Type":"ContainerStarted","Data":"ff65590cc61fc9dd602be162f3be08845e9f7bc05d08bee8ba1e933ad5274cd9"} Oct 30 18:36:48.719005 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.718963 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerStarted","Data":"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0"} Oct 30 18:36:48.719174 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.719053 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerStarted","Data":"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5"} Oct 30 18:36:48.719174 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.719094 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerStarted","Data":"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768"} Oct 30 18:36:48.720460 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.720436 2573 generic.go:356] "Generic (PLEG): container finished" podID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" exitCode=0 Oct 30 18:36:48.720557 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.720525 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerDied","Data":"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d"} Oct 30 18:36:48.720640 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.720598 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerStarted","Data":"71e42f68fd77314a41cf0580fb1c8ae2e24a30c25032800afd6bb05ad5da2786"} Oct 30 18:36:48.736622 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.736550 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" podStartSLOduration=1.872626687 podStartE2EDuration="3.736533199s" totalImagesPullingTime="1.863906512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:48.734945705 +0000 UTC m=+60.170446954" watchObservedRunningTime="2026-10-30 18:36:48.736533199 +0000 UTC m=+60.172034446" Oct 30 18:36:48.766651 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:48.766567 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.006656412 podStartE2EDuration="6.766551412s" totalImagesPullingTime="2.751440044s" totalInitContainerRuntime="8.454956ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:48.76495977 +0000 UTC m=+60.200461019" watchObservedRunningTime="2026-10-30 18:36:48.766551412 +0000 UTC m=+60.202052686" Oct 30 18:36:52.742488 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:52.742446 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerStarted","Data":"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626"} Oct 30 18:36:52.742925 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:52.742522 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerStarted","Data":"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0"} Oct 30 18:36:52.742925 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:52.742549 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerStarted","Data":"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c"} Oct 30 18:36:52.742925 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:52.742575 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerStarted","Data":"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0"} Oct 30 18:36:52.742925 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:52.742634 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerStarted","Data":"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2"} Oct 30 18:36:52.742925 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:52.742666 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerStarted","Data":"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d"} Oct 30 18:36:52.782238 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:52.782176 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.772520639 podStartE2EDuration="5.782157807s" totalImagesPullingTime="2.992939163s" totalInitContainerRuntime="16.698005ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:52.771475043 +0000 UTC m=+64.206976291" watchObservedRunningTime="2026-10-30 18:36:52.782157807 +0000 UTC m=+64.217659054" Oct 30 18:36:52.965121 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:52.965083 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:53.378182 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.378134 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:53.378364 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.378234 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:53.383139 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.383112 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:53.750549 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.750519 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:36:53.840904 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.840858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:53.844442 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.844415 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:36:53.853936 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.853910 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:53.941985 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.941942 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:53.941985 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.941986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:53.945368 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.945347 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:36:53.945426 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.945405 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:36:53.955704 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.955676 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:53.955962 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.955945 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:36:53.965753 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:53.965725 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:54.055507 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.055425 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnnzq\"" Oct 30 18:36:54.062657 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.062627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:36:54.073365 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.073344 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgk4h\"" Oct 30 18:36:54.075076 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.075006 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-phzbf\"" Oct 30 18:36:54.080109 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.080082 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:36:54.082178 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.082153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:54.344726 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.344700 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-628cx"] Oct 30 18:36:54.347176 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:54.347153 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d77554d_8094_4546_8fe7_0edf66752a5e.slice/crio-27916db1825d09aa9f0d5b6298dbd3b917abac7cfb5f0386047b0cb1cc85d0a9 WatchSource:0}: Error finding container 27916db1825d09aa9f0d5b6298dbd3b917abac7cfb5f0386047b0cb1cc85d0a9: Status 404 returned error can't find the container with id 27916db1825d09aa9f0d5b6298dbd3b917abac7cfb5f0386047b0cb1cc85d0a9 Oct 30 18:36:54.383668 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.383544 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qj49w"] Oct 30 18:36:54.386887 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:54.386859 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871c9271_d966_4c0d_b367_be51d1d4a79d.slice/crio-52d0b81c86626b6b09dcf955513939173bab68d93cca9207eda1c2d351f3d655 WatchSource:0}: Error finding container 52d0b81c86626b6b09dcf955513939173bab68d93cca9207eda1c2d351f3d655: Status 404 returned error can't find the container with id 52d0b81c86626b6b09dcf955513939173bab68d93cca9207eda1c2d351f3d655 Oct 30 18:36:54.388103 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.388081 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cbh4k"] Oct 30 18:36:54.390232 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:54.390210 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52d343a_4ee8_474d_b2ab_6fc96edde2fd.slice/crio-f0a7cbc3bb0733cbcba5fe23b17dfa277d8264ee049ecb08f9a7c19f35584a7b WatchSource:0}: Error finding container f0a7cbc3bb0733cbcba5fe23b17dfa277d8264ee049ecb08f9a7c19f35584a7b: Status 404 returned error can't find the container with id f0a7cbc3bb0733cbcba5fe23b17dfa277d8264ee049ecb08f9a7c19f35584a7b Oct 30 18:36:54.731334 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.731305 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:36:54.751249 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.751212 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cbh4k" event={"ID":"b52d343a-4ee8-474d-b2ab-6fc96edde2fd","Type":"ContainerStarted","Data":"f0a7cbc3bb0733cbcba5fe23b17dfa277d8264ee049ecb08f9a7c19f35584a7b"} Oct 30 18:36:54.752528 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.752503 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qj49w" event={"ID":"871c9271-d966-4c0d-b367-be51d1d4a79d","Type":"ContainerStarted","Data":"52d0b81c86626b6b09dcf955513939173bab68d93cca9207eda1c2d351f3d655"} Oct 30 18:36:54.753792 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:54.753768 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-628cx" event={"ID":"7d77554d-8094-4546-8fe7-0edf66752a5e","Type":"ContainerStarted","Data":"27916db1825d09aa9f0d5b6298dbd3b917abac7cfb5f0386047b0cb1cc85d0a9"} Oct 30 18:36:55.500136 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.498785 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-668d98c4ff-w5c7v"] Oct 30 18:36:55.512425 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.512383 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668d98c4ff-w5c7v"] Oct 30 18:36:55.512587 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.512502 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.523310 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.523263 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 18:36:55.557450 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.557408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.557450 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.557451 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.557717 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.557472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.557717 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.557516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqckm\" (UniqueName: \"kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.557717 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.557586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.557717 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.557637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.557717 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.557664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.557717 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.557688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.658455 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.658426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.658455 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.658466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.658705 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.658490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.658705 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.658512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.658705 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.658587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.658705 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.658640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.658705 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.658668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.658913 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.658734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqckm\" (UniqueName: \"kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.659456 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.659384 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.659947 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.659922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.660911 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.660535 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.660911 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.660881 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.661054 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.659922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.662211 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.662187 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.663502 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.663472 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.668592 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.668561 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqckm\" (UniqueName: \"kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:55.833130 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:55.832788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:36:56.146597 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:56.146557 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668d98c4ff-w5c7v"] Oct 30 18:36:56.152586 ip-10-0-133-96 kubenswrapper[2573]: W1030 18:36:56.152544 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671f39e7_0d53_41ac_af9a_750a01c38f71.slice/crio-2bf20d347dfa51b6449cf1355defaec52e74b130b6bd1a23a5c77e553d5d398f WatchSource:0}: Error finding container 2bf20d347dfa51b6449cf1355defaec52e74b130b6bd1a23a5c77e553d5d398f: Status 404 returned error can't find the container with id 2bf20d347dfa51b6449cf1355defaec52e74b130b6bd1a23a5c77e553d5d398f Oct 30 18:36:56.766247 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:56.765943 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qj49w" event={"ID":"871c9271-d966-4c0d-b367-be51d1d4a79d","Type":"ContainerStarted","Data":"815ad314f6f6e17c899f0c84d4d7df454bf6a0f29fa236c3ea04bcf97d1e16e6"} Oct 30 18:36:56.766247 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:56.766013 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qj49w" event={"ID":"871c9271-d966-4c0d-b367-be51d1d4a79d","Type":"ContainerStarted","Data":"b8dc5e4946a86b2a96ac080870aaba39dc194a38ec44048897b139d067d9fdf4"} Oct 30 18:36:56.767860 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:56.767833 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668d98c4ff-w5c7v" event={"ID":"671f39e7-0d53-41ac-af9a-750a01c38f71","Type":"ContainerStarted","Data":"7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde"} Oct 30 18:36:56.768004 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:56.767877 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668d98c4ff-w5c7v" event={"ID":"671f39e7-0d53-41ac-af9a-750a01c38f71","Type":"ContainerStarted","Data":"2bf20d347dfa51b6449cf1355defaec52e74b130b6bd1a23a5c77e553d5d398f"} Oct 30 18:36:56.811988 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:56.811926 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-668d98c4ff-w5c7v" podStartSLOduration=1.8119089659999998 podStartE2EDuration="1.811908966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:56.810735435 +0000 UTC m=+68.246236685" watchObservedRunningTime="2026-10-30 18:36:56.811908966 +0000 UTC m=+68.247410274" Oct 30 18:36:56.812286 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:56.812247 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qj49w" podStartSLOduration=66.556277293 podStartE2EDuration="1m7.812236146s" totalImagesPullingTime="1.255958853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:56.785842783 +0000 UTC m=+68.221344032" watchObservedRunningTime="2026-10-30 18:36:56.812236146 +0000 UTC m=+68.247737395" Oct 30 18:36:58.780042 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:58.780007 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cbh4k" event={"ID":"b52d343a-4ee8-474d-b2ab-6fc96edde2fd","Type":"ContainerStarted","Data":"1072f2b877f48687b9ccf49ee524c0d359cec4759e3761c5472166060fa49c9a"} Oct 30 18:36:58.780518 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:58.780146 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:36:58.796905 ip-10-0-133-96 kubenswrapper[2573]: I1030 18:36:58.796835 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cbh4k" podStartSLOduration=65.208249246 podStartE2EDuration="1m8.796819062s" totalImagesPullingTime="3.588569816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:58.796417075 +0000 UTC m=+70.231918324" watchObservedRunningTime="2026-10-30 18:36:58.796819062 +0000 UTC m=+70.232320309" Oct 30 18:36:59.607937 ip-10-0-133-96 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 18:36:59.621047 ip-10-0-133-96 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 18:36:59.621231 ip-10-0-133-96 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 18:36:59.621285 ip-10-0-133-96 systemd[1]: kubelet.service: Consumed 4.850s CPU time. Oct 30 18:36:59.636373 ip-10-0-133-96 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:36:59.703323 ip-10-0-133-96 kubenswrapper[5959]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:36:59.703323 ip-10-0-133-96 kubenswrapper[5959]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:36:59.703323 ip-10-0-133-96 kubenswrapper[5959]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:36:59.703323 ip-10-0-133-96 kubenswrapper[5959]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:36:59.705421 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705326 5959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:36:59.705421 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705421 5959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:36:59.705421 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705425 5959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705429 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705433 5959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705436 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705438 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705441 5959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705444 5959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705446 5959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705449 5959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705452 5959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705454 5959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705457 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705460 5959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705462 5959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705465 5959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705468 5959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705470 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705473 5959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705475 5959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705478 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:36:59.705518 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705481 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705483 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705485 5959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705488 5959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705491 5959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705497 5959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705500 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705503 5959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705505 5959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705509 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705511 5959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705514 5959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705517 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705520 5959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705523 5959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705526 5959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705528 5959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705531 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705534 5959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705537 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:36:59.706181 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705540 5959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705542 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705545 5959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705548 5959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705550 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705553 5959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705556 5959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705558 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705561 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705563 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705566 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705568 5959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705572 5959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705576 5959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705580 5959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705584 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705587 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705589 5959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705592 5959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:36:59.706904 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705594 5959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705615 5959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705618 5959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705621 5959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705623 5959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705628 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705630 5959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705633 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705636 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705638 5959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705641 5959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705643 5959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705646 5959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705648 5959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705651 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705654 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705656 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705659 5959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705662 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705664 5959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:36:59.707484 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705667 5959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705669 5959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705672 5959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705674 5959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705677 5959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705679 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705681 5959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705684 5959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705691 5959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705694 5959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705698 5959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705700 5959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705703 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705707 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705710 5959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705712 5959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705715 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705718 5959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705720 5959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:36:59.707988 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705723 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705725 5959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705729 5959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705731 5959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705734 5959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.705737 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706563 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706572 5959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706575 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706578 5959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706581 5959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706583 5959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706586 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706588 5959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706591 5959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706594 5959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706596 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706599 5959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706620 5959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706625 5959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:36:59.708440 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706629 5959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706634 5959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706638 5959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706641 5959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706646 5959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706650 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706655 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706658 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706661 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706664 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706667 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706669 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706672 5959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706675 5959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706677 5959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706680 5959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706683 5959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706686 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706688 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:36:59.708963 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706691 5959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706693 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706696 5959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706698 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706701 5959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706704 5959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706706 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706709 5959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706711 5959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706713 5959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706716 5959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706718 5959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706721 5959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706723 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706726 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706728 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706732 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706734 5959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706737 5959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:36:59.709408 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706742 5959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706745 5959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706748 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706751 5959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706753 5959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706756 5959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706758 5959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706761 5959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706763 5959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706766 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706768 5959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706771 5959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706774 5959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706777 5959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706780 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706782 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706785 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706788 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706790 5959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:36:59.709881 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706793 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706795 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706798 5959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706800 5959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706803 5959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706805 5959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706808 5959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706811 5959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706814 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706817 5959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706819 5959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706822 5959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706825 5959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706829 5959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706832 5959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706835 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706837 5959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706840 5959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706842 5959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:36:59.710327 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706845 5959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706847 5959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706850 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706853 5959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706855 5959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706858 5959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706861 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706863 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706866 5959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706868 5959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706871 5959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706874 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706877 5959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706883 5959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706887 5959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.706889 5959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707471 5959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707480 5959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707483 5959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:36:59.710960 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707488 5959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707492 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707496 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707499 5959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707502 5959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707505 5959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707508 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707511 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707514 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707516 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707519 5959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707522 5959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707524 5959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707527 5959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707529 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707532 5959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707534 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707537 5959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707539 5959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707541 5959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:36:59.711700 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707544 5959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707546 5959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707549 5959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707551 5959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707554 5959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707557 5959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707559 5959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707562 5959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707564 5959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707567 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707570 5959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707572 5959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707575 5959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707577 5959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707580 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707583 5959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707585 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707588 5959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707591 5959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707594 5959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:36:59.712539 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707597 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707599 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707620 5959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707624 5959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707628 5959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707631 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707634 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707637 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707640 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707642 5959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707645 5959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707647 5959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707650 5959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707652 5959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707655 5959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707658 5959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707660 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707665 5959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707668 5959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:36:59.713410 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707671 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707673 5959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707676 5959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707678 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707681 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707684 5959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707686 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707689 5959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707691 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707694 5959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707698 5959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707702 5959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707706 5959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707709 5959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707712 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707715 5959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707717 5959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707720 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707722 5959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:36:59.714230 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707725 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707728 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707730 5959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707733 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707735 5959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707738 5959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707740 5959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707742 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707745 5959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707747 5959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707750 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707753 5959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707756 5959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707758 5959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707761 5959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707763 5959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707766 5959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707768 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707771 5959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:36:59.715053 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707774 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707777 5959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707781 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707784 5959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707786 5959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.707789 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707891 5959 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707899 5959 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707906 5959 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707913 5959 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707920 5959 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707924 5959 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707929 5959 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707934 5959 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707937 5959 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707940 5959 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707944 5959 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707947 5959 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707950 5959 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707953 5959 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707956 5959 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707959 5959 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707962 5959 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:36:59.715911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707965 5959 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707970 5959 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707973 5959 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707976 5959 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707979 5959 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707982 5959 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707986 5959 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707990 5959 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707993 5959 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.707998 5959 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708001 5959 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708004 5959 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708007 5959 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708010 5959 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708013 5959 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708018 5959 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708021 5959 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708025 5959 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708046 5959 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708050 5959 flags.go:64] FLAG: --enable-server="true" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708053 5959 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708058 5959 flags.go:64] FLAG: --event-burst="100" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708061 5959 flags.go:64] FLAG: --event-qps="50" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708064 5959 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:36:59.716928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708068 5959 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708071 5959 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708075 5959 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708078 5959 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708081 5959 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708084 5959 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708087 5959 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708090 5959 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708093 5959 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708098 5959 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708100 5959 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708103 5959 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708106 5959 flags.go:64] FLAG: --feature-gates="" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708110 5959 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708113 5959 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708116 5959 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708119 5959 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708125 5959 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708128 5959 flags.go:64] FLAG: --help="false" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708131 5959 flags.go:64] FLAG: --hostname-override="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708134 5959 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708137 5959 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708140 5959 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:36:59.718057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708143 5959 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708147 5959 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708150 5959 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708153 5959 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708156 5959 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708159 5959 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708162 5959 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708165 5959 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708168 5959 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708171 5959 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708173 5959 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708176 5959 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708179 5959 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708182 5959 flags.go:64] FLAG: --lock-file="" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708185 5959 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708188 5959 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708191 5959 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708196 5959 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708200 5959 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708203 5959 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708206 5959 flags.go:64] FLAG: --logging-format="text" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708208 5959 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708211 5959 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708214 5959 flags.go:64] FLAG: --manifest-url="" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708217 5959 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:36:59.718642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708222 5959 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708226 5959 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708230 5959 flags.go:64] FLAG: --max-pods="110" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708233 5959 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708236 5959 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708239 5959 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708242 5959 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708245 5959 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708248 5959 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708251 5959 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708258 5959 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708262 5959 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708265 5959 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708269 5959 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708272 5959 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708275 5959 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708277 5959 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708280 5959 flags.go:64] FLAG: --port="10250" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708283 5959 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708286 5959 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08f6538b272b50780" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708289 5959 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708292 5959 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708295 5959 flags.go:64] FLAG: --register-node="true" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708298 5959 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708302 5959 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708306 5959 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:36:59.719282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708309 5959 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708312 5959 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708316 5959 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708319 5959 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708322 5959 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708325 5959 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708328 5959 flags.go:64] FLAG: --runonce="false" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708332 5959 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708335 5959 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708338 5959 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708341 5959 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708344 5959 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708348 5959 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708350 5959 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708354 5959 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708357 5959 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708360 5959 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708363 5959 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708373 5959 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708377 5959 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708380 5959 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708383 5959 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708389 5959 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708392 5959 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708394 5959 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:36:59.720002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708398 5959 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708401 5959 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708404 5959 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708406 5959 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708409 5959 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708412 5959 flags.go:64] FLAG: --v="2" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708422 5959 flags.go:64] FLAG: --version="false" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708427 5959 flags.go:64] FLAG: --vmodule="" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708430 5959 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708434 5959 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708582 5959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708585 5959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708588 5959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708591 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708595 5959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708598 5959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708618 5959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708621 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708624 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708627 5959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708629 5959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708632 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:36:59.720677 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708634 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708637 5959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708639 5959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708642 5959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708645 5959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708648 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708650 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708653 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708656 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708658 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708661 5959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708663 5959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708666 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708668 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708671 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708673 5959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708677 5959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708679 5959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708681 5959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708684 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:36:59.721203 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708687 5959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708691 5959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708694 5959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708697 5959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708704 5959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708707 5959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708709 5959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708713 5959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708717 5959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708721 5959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708724 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708727 5959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708730 5959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708732 5959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708735 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708738 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708740 5959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708743 5959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:36:59.721772 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708746 5959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708748 5959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708751 5959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708754 5959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708756 5959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708759 5959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708761 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708764 5959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708766 5959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708769 5959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708771 5959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708774 5959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708776 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708779 5959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708781 5959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708784 5959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708786 5959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708789 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708793 5959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708796 5959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:36:59.722313 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708798 5959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708801 5959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708803 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708806 5959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708808 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708810 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708813 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708815 5959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708818 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708820 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708823 5959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708826 5959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708829 5959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708832 5959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708834 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708837 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708839 5959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708842 5959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708844 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:36:59.723025 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708847 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708849 5959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708852 5959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708854 5959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708857 5959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708859 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708862 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708864 5959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708867 5959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708869 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708872 5959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708875 5959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708878 5959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708880 5959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708883 5959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708886 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:36:59.723692 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708888 5959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.708915 5959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708954 5959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708959 5959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708962 5959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708965 5959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708967 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708970 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708973 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708976 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708979 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708981 5959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708984 5959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708986 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708989 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708991 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:36:59.724164 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708994 5959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708996 5959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.708999 5959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709001 5959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709004 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709006 5959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709010 5959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709015 5959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709018 5959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709021 5959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709024 5959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709026 5959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709031 5959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709035 5959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709038 5959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709041 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709043 5959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709046 5959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709048 5959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:36:59.724686 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709051 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709054 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709056 5959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709059 5959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709061 5959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709064 5959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709067 5959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709069 5959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709072 5959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709075 5959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709077 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709080 5959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709082 5959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709085 5959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709087 5959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709090 5959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709092 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709094 5959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709097 5959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709099 5959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:36:59.725208 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709103 5959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709105 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709108 5959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709111 5959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709113 5959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709116 5959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709119 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709121 5959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709124 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709126 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709129 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709131 5959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709134 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709136 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709139 5959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709141 5959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709144 5959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709147 5959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709150 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709152 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:36:59.725908 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709155 5959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709158 5959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709161 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709163 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709166 5959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709168 5959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709171 5959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709173 5959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709175 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709178 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709180 5959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709183 5959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709187 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709189 5959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709192 5959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709194 5959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709197 5959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709200 5959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709203 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709205 5959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:36:59.726575 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709208 5959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709210 5959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709213 5959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709215 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709218 5959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709220 5959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709223 5959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709226 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709229 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709232 5959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709235 5959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709237 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.709240 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.714780 5959 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.714804 5959 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.714830 5959 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.714838 5959 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714944 5959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714953 5959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:36:59.727252 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714959 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714964 5959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714968 5959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714973 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714978 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714982 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714986 5959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714990 5959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714995 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.714999 5959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715011 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715015 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715020 5959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715024 5959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715028 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715035 5959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715042 5959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715047 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715052 5959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715056 5959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:36:59.727888 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715060 5959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715063 5959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715067 5959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715071 5959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715075 5959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715081 5959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715085 5959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715090 5959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715094 5959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715099 5959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715103 5959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715108 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715112 5959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715116 5959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715120 5959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715124 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715128 5959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715132 5959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715137 5959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:36:59.728389 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715141 5959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715145 5959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715149 5959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715153 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715168 5959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715172 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715177 5959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715181 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715185 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715189 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715193 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715206 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715211 5959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715215 5959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715219 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715223 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715227 5959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715231 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715239 5959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:36:59.728863 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715246 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715253 5959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715259 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715264 5959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715270 5959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715276 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715280 5959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715284 5959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715289 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715293 5959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715297 5959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715301 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715305 5959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715309 5959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715313 5959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715318 5959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715322 5959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715326 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715338 5959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715343 5959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:36:59.729347 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715347 5959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715351 5959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715355 5959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715360 5959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715364 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715368 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715373 5959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715377 5959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715381 5959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715385 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715389 5959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715393 5959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715398 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715402 5959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715406 5959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715410 5959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715414 5959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715418 5959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715423 5959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715428 5959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:36:59.729851 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715432 5959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715436 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715440 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715444 5959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715449 5959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715453 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.715498 5959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715816 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715834 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715840 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715845 5959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715857 5959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715862 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715866 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715870 5959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715875 5959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:36:59.730353 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715879 5959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715883 5959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715887 5959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715891 5959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715896 5959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715900 5959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715904 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715908 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715912 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715916 5959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715920 5959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715924 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715928 5959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715932 5959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715937 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715941 5959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715946 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715950 5959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715954 5959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715958 5959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:36:59.730901 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715962 5959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715966 5959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715971 5959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715975 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715979 5959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715983 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715988 5959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.715992 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716004 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716008 5959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716013 5959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716017 5959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716020 5959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716025 5959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716029 5959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716033 5959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716037 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716041 5959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716045 5959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:36:59.731562 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716049 5959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716056 5959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716061 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716066 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716070 5959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716075 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716079 5959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716084 5959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716088 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716093 5959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716097 5959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716102 5959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716105 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716109 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716114 5959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716118 5959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716122 5959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716127 5959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716131 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716135 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:36:59.732247 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716139 5959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716143 5959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716227 5959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716452 5959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716459 5959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716464 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716469 5959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716474 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716479 5959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716483 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716488 5959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716492 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716496 5959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716501 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716505 5959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716518 5959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716522 5959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716527 5959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716531 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:36:59.732917 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716536 5959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716540 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716544 5959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716549 5959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716554 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716559 5959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716563 5959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716568 5959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716578 5959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716583 5959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716587 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716592 5959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716617 5959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716627 5959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716632 5959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716636 5959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716686 5959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716691 5959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:36:59.733506 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716696 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.716768 5959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716953 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716962 5959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716968 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716973 5959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716978 5959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716982 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716992 5959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.716997 5959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717001 5959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717006 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717010 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717015 5959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717019 5959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717025 5959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:36:59.734072 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717030 5959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717034 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717039 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717043 5959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717047 5959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717058 5959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717063 5959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717067 5959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717072 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717076 5959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717080 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717085 5959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717089 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717094 5959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717098 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717135 5959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717141 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717146 5959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717155 5959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:36:59.734577 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717160 5959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717164 5959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717169 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717173 5959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717178 5959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717182 5959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717187 5959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717191 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717196 5959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717200 5959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717204 5959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717209 5959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717219 5959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717226 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717231 5959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717237 5959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717244 5959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717249 5959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717254 5959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717259 5959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:36:59.735225 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717264 5959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717269 5959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717274 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717279 5959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717289 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717294 5959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717301 5959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717305 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717309 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.717827 5959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718001 5959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718009 5959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718015 5959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718019 5959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718023 5959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718028 5959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718032 5959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718036 5959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718040 5959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:36:59.735975 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718045 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718049 5959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718053 5959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718057 5959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718061 5959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718066 5959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718071 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718075 5959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718079 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718083 5959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718088 5959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718092 5959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718096 5959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718101 5959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718104 5959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718108 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718112 5959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718116 5959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718123 5959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:36:59.736593 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718127 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718132 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718138 5959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718142 5959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718147 5959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718151 5959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718155 5959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718159 5959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718163 5959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718170 5959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718174 5959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718178 5959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718182 5959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718191 5959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:36:59.718198 5959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.718308 5959 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.719432 5959 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.719503 5959 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 18:36:59.737245 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.719722 5959 server.go:1014] "Starting client certificate rotation" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.719866 5959 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.719927 5959 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:30:49 +0000 UTC" deadline="2028-08-11 10:52:38.644881332 +0000 UTC" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.719968 5959 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15616h15m38.924916754s" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.720245 5959 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.721553 5959 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.723048 5959 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.725649 5959 log.go:25] "Validated CRI v1 image API" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.726530 5959 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:36:59.737952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.729308 5959 fs.go:135] Filesystem UUIDs: map[256dd621-7796-4e21-8cce-9866d8d51c19:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a7af5301-f3e2-4eb8-9d5f-dc0e5ab8e9f5:/dev/nvme0n1p4] Oct 30 18:36:59.738350 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.729330 5959 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04cdbf66861fa622868c7baf6998d9d45d6466ac731f195cb70655c86f6a3619/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04cdbf66861fa622868c7baf6998d9d45d6466ac731f195cb70655c86f6a3619/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27916db1825d09aa9f0d5b6298dbd3b917abac7cfb5f0386047b0cb1cc85d0a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27916db1825d09aa9f0d5b6298dbd3b917abac7cfb5f0386047b0cb1cc85d0a9/userdata/shm major:0 minor:495 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bf20d347dfa51b6449cf1355defaec52e74b130b6bd1a23a5c77e553d5d398f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bf20d347dfa51b6449cf1355defaec52e74b130b6bd1a23a5c77e553d5d398f/userdata/shm major:0 minor:518 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4020703a95484315726b9502ccaafd08d26dc3e0287b28771549d24120e66621/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4020703a95484315726b9502ccaafd08d26dc3e0287b28771549d24120e66621/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/433f3b2041f96c84031e68c7d26bdf986bda69de52d983aaa7b3f97bc2f6f05c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/433f3b2041f96c84031e68c7d26bdf986bda69de52d983aaa7b3f97bc2f6f05c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/508bd618fe005a261db3945088046ea1b6101b237b62bc1eb1db72e982437d00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/508bd618fe005a261db3945088046ea1b6101b237b62bc1eb1db72e982437d00/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52d0b81c86626b6b09dcf955513939173bab68d93cca9207eda1c2d351f3d655/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52d0b81c86626b6b09dcf955513939173bab68d93cca9207eda1c2d351f3d655/userdata/shm major:0 minor:497 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c4196f8503be0d4a88dda1f4100de07bf9130710453e0c72bbd866f520390b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c4196f8503be0d4a88dda1f4100de07bf9130710453e0c72bbd866f520390b0/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/662790f5602c3adfc5b440570e9c38e9e9690b27315e25f6bb06ba30956a47df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/662790f5602c3adfc5b440570e9c38e9e9690b27315e25f6bb06ba30956a47df/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/676128efd1ec3d1cabda721eefbfe19ede0f6625081a57a8ac54f1482ee85f68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/676128efd1ec3d1cabda721eefbfe19ede0f6625081a57a8ac54f1482ee85f68/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fbcb57cf18754b006c8d806fd1e4164b7234bc19cd69677f3841ceb9bee960e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fbcb57cf18754b006c8d806fd1e4164b7234bc19cd69677f3841ceb9bee960e/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71e42f68fd77314a41cf0580fb1c8ae2e24a30c25032800afd6bb05ad5da2786/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71e42f68fd77314a41cf0580fb1c8ae2e24a30c25032800afd6bb05ad5da2786/userdata/shm major:0 minor:448 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a2c398fa990b15181a4e101f345ddf18c076806db928e3f9d01c1dbbb37d7eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a2c398fa990b15181a4e101f345ddf18c076806db928e3f9d01c1dbbb37d7eb/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7acf902a75afca9f4419ca945c7c017a5fe82b3cefb108cab39234025d4c4f25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7acf902a75afca9f4419ca945c7c017a5fe82b3cefb108cab39234025d4c4f25/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b42b15eb9b8d77cc4bcdc91e1f36a8c3e86bede2d68f98a109038d20725dd9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b42b15eb9b8d77cc4bcdc91e1f36a8c3e86bede2d68f98a109038d20725dd9c/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a06c77803ed1a18115d666293b30bb15c5045a523b6b6ccdedae5ae879e71cde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a06c77803ed1a18115d666293b30bb15c5045a523b6b6ccdedae5ae879e71cde/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0e139b70bb14fa05239c9166e60b22218a5eb30760ad8a4d832a8c726af2cf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0e139b70bb14fa05239c9166e60b22218a5eb30760ad8a4d832a8c726af2cf7/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9e9f60d6c2931c6b6f0f7b6523a3f4e91f39eb6834d1edf73fe15007a760d1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9e9f60d6c2931c6b6f0f7b6523a3f4e91f39eb6834d1edf73fe15007a760d1a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb627f0ff19dec1f579603dfb93d7d049ce7e150e4f7cde6802fa6d4ab735564/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb627f0ff19dec1f579603dfb93d7d049ce7e150e4f7cde6802fa6d4ab735564/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be0307adac5f8682c13f30b6c44ae7df5217989fe5a8d87c1a9ee18ccc4ca2d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be0307adac5f8682c13f30b6c44ae7df5217989fe5a8d87c1a9ee18ccc4ca2d1/userdata/shm major:0 minor:58 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfb9e46b67dc8a2b747d12243c10961fa20445e870fcb4da4157f88ccfe476cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfb9e46b67dc8a2b747d12243c10961fa20445e870fcb4da4157f88ccfe476cf/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d43a365bed980790e8a1fa057ee1e434d810db1209c724fd4f17c7c67e9da874/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d43a365bed980790e8a1fa057ee1e434d810db1209c724fd4f17c7c67e9da874/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6e21717adb75fa3fdf56f43294b1cb5389c0a5ffa9280c7ce635560fedcb2a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6e21717adb75fa3fdf56f43294b1cb5389c0a5ffa9280c7ce635560fedcb2a8/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0d1b0b56f4aa1b164fe8d1d0676e21f5355ddd50d625d3e9809d384e39bfab0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0d1b0b56f4aa1b164fe8d1d0676e21f5355ddd50d625d3e9809d384e39bfab0/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0a7cbc3bb0733cbcba5fe23b17dfa277d8264ee049ecb08f9a7c19f35584a7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0a7cbc3bb0733cbcba5fe23b17dfa277d8264ee049ecb08f9a7c19f35584a7b/userdata/shm major:0 minor:498 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa09d3e6f1b46069f3cef0db14329bda4be2d2952341a287404ce35edb72f55c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa09d3e6f1b46069f3cef0db14329bda4be2d2952341a287404ce35edb72f55c/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272dcf18-8186-45d9-9316-8c5a0efc951b/volumes/kubernetes.io~projected/kube-api-access-qhmql:{mountpoint:/var/lib/kubelet/pods/272dcf18-8186-45d9-9316-8c5a0efc951b/volumes/kubernetes.io~projected/kube-api-access-qhmql major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/353c68dc-e91f-445b-b4f0-386cf1a9c8e7/volumes/kubernetes.io~projected/kube-api-access-bcqjj:{mountpoint:/var/lib/kubelet/pods/353c68dc-e91f-445b-b4f0-386cf1a9c8e7/volumes/kubernetes.io~projected/kube-api-access-bcqjj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/353c68dc-e91f-445b-b4f0-386cf1a9c8e7/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/353c68dc-e91f-445b-b4f0-386cf1a9c8e7/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3/volumes/kubernetes.io~projected/kube-api-access-nwdpg:{mountpoint:/var/lib/kubelet/pods/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3/volumes/kubernetes.io~projected/kube-api-access-nwdpg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/alertmanager/9 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/config-reloader/6 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~empty-dir/config-out major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~projected/kube-api-access-vsfb8:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~projected/kube-api-access-vsfb8 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~projected/tls-assets major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/config-volume major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/web-config major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~projected/kube-api-access-wqckm:{mountpoint:/var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~projected/kube-api-access-wqckm major:0 minor:511 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:507 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:508 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~projected/kube-api-access-zqmp8:{mountpoint:/var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~projected/kube-api-access-zqmp8 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d77554d-8094-4546-8fe7-0edf66752a5e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7d77554d-8094-4546-8fe7-0edf66752a5e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:493 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/config-reloader/2 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/prometheus/15 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~empty-dir/config-out major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~projected/kube-api-access-p5lf6:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~projected/kube-api-access-p5lf6 major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~projected/tls-assets major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/config major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/web-config major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/871c9271-d966-4c0d-b367-be51d1d4a79d/volumes/kubernetes.io~projected/kube-api-access-btts9:{mountpoint:/var/lib/kubelet/pods/871c9271-d966-4c0d-b367-be51d1d4a79d/volumes/kubernetes.io~projected/kube-api-access-btts9 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/871c9271-d966-4c0d-b367-be51d1d4a79d/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/871c9271-d966-4c0d-b367-be51d1d4a79d/volumes/kubernetes.io~secret/metrics-certs major:0 minor:489 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89a132e7-7020-4c9e-accd-89b0a61d73b0/volumes/kubernetes.io~projected/kube-api-access-5n855:{mountpoint:/var/lib/kubelet/pods/89a132e7-7020-4c9e-accd-89b0a61d73b0/volumes/kubernetes.io~projected/kube-api-access-5n855 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d86b7d9-2e05-4f95-b002-931249ab66e2/volumes/kubernetes.io~projected/kube-api-access-8d52x:{mountpoint:/var/lib/kubelet/pods/8d86b7d9-2e05-4f95-b002-931249ab66e2/volumes/kubernetes.io~projected/kube-api-access-8d52x major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d86b7d9-2e05-4f95-b002-931249ab66e2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8d86b7d9-2e05-4f95-b002-931249ab66e2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~projected/kube-api-access-b5x6c:{mountpoint:/var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~projected/kube-api-access-b5x6c major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~projected/kube-api-access-hn6dx:{mountpoint:/var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~projected/kube-api-access-hn6dx major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volumes/kubernetes.io~projected/kube-api-access-jw9k9:{mountpoint:/var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volumes/kubernetes.io~projected/kube-api-access-jw9k9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~projected/kube-api-access-5xb5h:{mountpoint:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~projected/kube-api-access-5xb5h major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b52d343a-4ee8-474d-b2ab-6fc96edde2fd/volumes/kubernetes.io~projected/kube-api-access-bv2pn:{mountpoint:/var/lib/kubelet/pods/b52d343a-4ee8-474d-b2ab-6fc96edde2fd/volumes/kubernetes.io~projected/kube-api-access-bv2pn major:0 minor:494 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8994e33-9b53-41d7-8fa8-04463a3ae647/volumes/kubernetes.io~projected/kube-api-access-q5zg5:{mountpoint:/var/lib/kubelet/pods/c8994e33-9b53-41d7-8fa8-04463a3ae647/volumes/kubernetes.io~projected/kube-api-access-q5zg5 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc878e8c-7351-4591-9939-4d33d63b6d1e/volumes/kubernetes.io~projected/kube-api-access-zxh5h:{mountpoint:/var/lib/kubelet/pods/cc878e8c-7351-4591-9939-4d33d63b6d1e/volumes/kubernetes.io~projected/kube-api-access-zxh5h major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce5ba25c-da20-40d5-b02f-89e94aaad0d2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ce5ba25c-da20-40d5-b02f-89e94aaad0d2/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~projected/kube-api-access-wpbdq:{mountpoint:/var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~projected/kube-api-access-wpbdq major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~projected/kube-api-access-v2b6n:{mountpoint:/var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~projected/kube-api-access-v2b6n major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9dc8719-07a1-41fe-834b-66bbd4e9e19b/volumes/kubernetes.io~projected/kube-api-access-chwjc:{mountpoint:/var/lib/kubelet/pods/d9dc8719-07a1-41fe-834b-66bbd4e9e19b/volumes/kubernetes.io~projected/kube-api-access-chwjc major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~projected/kube-api-access-qtcjs:{mountpoint:/var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~projected/kube-api-access-qtcjs major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~projected/kube-api-access-9nvxp:{mountpoint:/var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~projected/kube-api-access-9nvxp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbbbc1ad-2bb2-469a-8b29-94760bce83cc/volumes/kubernetes.io~projected/kube-api-access-q8hs8:{mountpoint:/var/lib/kubelet/pods/fbbbc1ad-2bb2-469a-8b29-94760bce83cc/volumes/kubernetes.io~projected/kube-api-access-q8hs8 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbbbc1ad-2bb2-469a-8b29-94760bce83cc/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/fbbbc1ad-2bb2-469a-8b29-94760bce83cc/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5deaa7b01d740358b15507c0e19ecf6d174ca361f83afab56366b4aedb96060b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/22a4e0e9768012c89d9243d151a3140e5613fa97e72b306b64673f8e212d32a1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8367df513dc1036ac3b5b45339962bbc47d5a6f4369d33734caf91fb9c5bf827/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8d99e0033bcf81f1d110e43292adfc416662f67e74e306156fc813415982557f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cc393a4af44a442a0daab3bc63d7f776459186678cc87692d411fd1146298134/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ee1c22f14addd76199a50879a9a55aa074bd4cf2fd3226ef343cf5694206614c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/342c01b33b7623b781efea062671475b7c7402d7bae8cefec9129bace9b82801/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5309ee9d2b738da3e68c28d92b1905932b70d5c515e100aeb8e9c9e36df396bc/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/f23800be3da0079143f6c56e081f0ee50ec369cc16c6793ce797ae447cdfdb1e/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/094fb2b2549f177751f79be63408dcad530a087ed79aac969422ecd613b63644/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/362d909289230dd3ac58c700e7fe8f66c570541cc915a448b0979afb1c084a14/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/0e39fdfa5454a8db3884eefef8d60eb9b6459cccd1722f29729ca7ee0219a5c0/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6d973487c4bcba4a2c8338417dbcb6757f92272be9dd6a219c610ba4b2a5a474/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9ffdba373aa3ef63911a45a2bce9b056227fa49de775b26a87e9860f159529cf/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7254e97c15752032407ec0f32c81bac2da65470159e0c4e9a42e4141124f0b53/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6c661bebc9aceb7e429256608ed019a4d00598ebce1d37440814ed37e7ba290b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c44cfdcb6c555cb9a2692f1b94472fcf97757d21133b370df2ee0fddeebc01a0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fe0ee6e441db25c1947bc0053071a513e7611857bfca5df4398470a98de71426/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/409402d76782841af126600e4bcaf4efdfbac0494fbc3439db86c20343221124/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9cc400b4403ef49913e2fb02d570e30f25e8982d336da08a1f81217f79d534d9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e73004a52764cf4fb564a75c89012e02b27d56123c56a45eb7ee6a2d1ede7bc0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/8fa0e21177c854fed9b48d97e437ec6b511582bd8fc42e12082514593d0ac392/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/22c6ffc90c37b93ef7ef1f056aac0b6934352178f40013ca7b624282c2ee2144/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/57fc51e0f2a8f47cf17aa51ba2e103df04f581df6814b754d695055b4bc385eb/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7995da0c7bdfa2e3c3568adda218933e1143bf7cbe080ab6d360ba36bf64d9b2/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/821e5ecee08d702d79ccbb889bd3b0d1830298d9d0b2517f8969dda63c942c74/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/170a90b2730cde104aed2ea04d32b3a5410984390312934d02932157d4f8d366/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/efa1960a26726a43eb9aeba0a6c1fa563d9d9355f9dc0acbae5926b5ebdc44bb/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/d4cb73a106105388caccb4988695d88689cf155029bbf416761f2b5a4b9a5638/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/56158dc47f83e3aa294d158eaf85551ef15c595e7ce5dd5027856a4536e3f47b/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/092027030e5dfb77a8ca411d54365bae10d836fdb4120225ba967d5d8df96219/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/bdb5de79a7535d64737e48a957183d4172944645d06d70bf4f6a523f29495c51/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/53ba769af37e4e7a22678f8b889a5e885c57c4892bcb8c8d9f5301c58a914151/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/b9e06837888a5a0acecc2640749a1d140f6c7b2a10303de79e3dd822930b5a62/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/1651837a2729afca07cd97870f2d5cf97f7cf3292a47b1e6fbf0b70e8708ef06/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/66f0dfc938c2aea60cdbf4e3d1c0876a7814100541e0a296da58a85d9eb09887/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/c0f88f3b436e76e7a05895a02306249875934dd6c261ff9d5e05600d0bc287a9/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/73977254ee08bd8bcdb250b03f55630f8369badcdbd3fefe0e2f52360bf74fb9/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/362f2c51552092012628676166c7ec4896d4409169a2a35af615e6fd5c0d883e/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/3e2419f32972aa814e5cb42159ba2c4eb8d86c5438ff4293e8db5c031ba32a37/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/3f010027a8ea33008b0e161112d0f7ef62321ca0d1eee69d682d4d22c9ae7e05/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/1924a944dee507c090900bdba07d00b96558eec869016492e365e0fb32feaf27/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/2a0f37f4cfe3abafbb962c615b48398b2db2d6d2d3f3ff366ac43e8fae7c0740/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/4293e119d06d686fab7ddc717a6ce0e0b2b56079ebb0fd4cf32685425fbf030b/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/a9b2b319cfa1abc002981a71c7ea001ce9dd8eb942546fd67ef3f5dcb2609e10/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/bf92d15bc690d5cf22df612f22fcdbda8f3d8c8a9dae48693eba82428216de48/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/f72215f6c863cc6d33c50a6804526faf893f8b16746e507aea1f5c062b46503e/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/b956a260530b6553cb8b63c104d03c04fe6aaaa6f703e8bba1295099164d544b/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/57df6edea0f1910c68f16338d7edea73c96175a80e09abafff473e0b883c078f/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/7271e014fe8a0332424f05a0f338e75be971864db9ad8fe3c1fbff92c790bfed/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/f9e3f47bfbaa22ff039db14690ee496100f228ea02b3451d71d647c69dbf8897/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/4cd388fe6bc2eec6dc49953f0c0f9d86a5c41f3e69a4983a4b15a730c3573563/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/260a7c684e91c174b5e14ecabdfc86d2d456752f8022573116808241bd480e76/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/9d438646482c51dc8c913d764494590d4421112a087bdc20d3faae82772737d7/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/22ecae5ef10d2dfeeac871f1973aababe55003ac021f7a5afeccdb89e3e6f028/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/1892021ef9a7354a20e7fe01d0f0ce5d907e7dd6247d9294f0e240d18f973c44/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/bd4486137ccb431c6c4cc365d168f4b2f96e2dd3718ad4909aeb31ba24f07b58/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/e3015e0b9ac3597da9212fe5b2c1145beaca9cdb7af8cb48a24d6c1fdeaf63dd/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/b0de87b16489624d1d8dea6453adf519756276d262e29566f064787ba1dbd293/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/3a06c0f3677afc4d3e74b4fa25fb4ee999864c9d5d0dc92dafab37ae9a87fb3e/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/992df0f1cfd76059976e2a322646c7454111d6c98264d6621125adbfd8fb8d75/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/a40dd5c10c2966f99a5528a008775116f08cb7dd145756bf415ca09782c7b2bd/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/a0b26da2d8c1e20b900f3181b1b99075676aba2977c010e07d84948660f671c4/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/1d2c5e12f551745453a5a01261377ddc40e8b445a3684ac1d562555e83f4a19f/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/3f772d3ca0b7c8fdbc157d7db9456fe00841ad18900010f43ad8bc3ab2e41b78/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/f636417e434e37aaf1a1ae433c458d736bdbde1ac24dbc92aab469591c0acd02/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/797f061019b93f3039b7413089854989b6e0f405451112a1de82e2824844da23/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-479:{mountpoint:/var/lib/containers/storage/overlay/e61673d71768f3bb2d3ab1d342da547bbcf547f2464c555ef4d48a5882e65686/merged major:0 minor:479 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/7d3240ac3356abd0d8429ff3458b6d32e00d4d4ebd26f6511971cddc1b1d15d6/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/d1d0fa516642d57ed74bda8b4df20616304628b977062e781fe5853ef5611f26/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/1ed3e37f12eae341b825f3519cce1e00733926e05620c8a9edbb255ac3d0f9ae/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/8071489c91bec35b248ea5a4026b463f87b1c63ed17e5702654c017274ee719f/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/f47368b943c2dfc76c20666056fe254f066405ebaff059ec1e8e4acd0cd165b6/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-512:{mountpoint:/var/lib/containers/storage/overlay/90a89ef0709270b8b748382b8d2051d44217bd38d69433e29426f46ea71b74fa/merged major:0 minor:512 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/bc2017f8399745989f4ea823dcce104a473bbe63b9e98e614c2f1dc1840ed638/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-525:{mountpoint:/var/lib/containers/storage/overlay/54e9a8c98a78be95e58bd92cd4b539c50dafc7e2197d9b580b2b581c369e3463/merged major:0 minor:525 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/18c04d241c93eb5e13a714bfae2df9b634b0c957925fe3663032ff37ddadfe95/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-533:{mountpoint:/var/lib/containers/storage/overlay/bf504967e2bba4d42cb5e270eae3a895a0a988c7aa9eaf79f2c36496ba79db5c/merged major:0 minor:533 fsType:overlay blockSize:0} overlay_0-61:{mountpoint:/var/lib/containers/storage/overlay/3dca73525cdc5e50504a65aae3d9c773d43a4cc12534bb09f41d4d7f443e0e97/merged major:0 minor:61 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/85b356308fe95292aa58c15dc8e38d92842b13cf97cb3dfe5a1cae39b27a8d7b/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/fecdb80e049a39525773c0e1b052596c0ae8755ac2ff5f364614db0e6cb098d9/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/133c343f95894057b687aa598ff2e3b257f163b2671078eb9e7be0c9e323a7ee/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/1e05a6e4c2294720ec4bc2afa915b66704dc7ba53afdc37812d46d3c853e4c02/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/1d78f083d7e5cb116deb4a24cd4a2669df656a6b25dd4b7acf775310560c6937/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/08fda49ec69974f151cd0b30ba0f22727545cd124afbcf57273aa6d3cda96632/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4eefb4733b7680e7152570a495f9421becf923331b86ef2286d1c89ea197f707/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/3bfdc8f51fe9642b4ed7ecf6bdac5ed99c872c6c854a618fe595a6194f872163/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/12ae7bfc46006a491a1ad62384fa6f8da3d13d71368836f0f53ce6f68bfa322a/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 18:36:59.740101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.738975 5959 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:59.738141899 +0000 UTC m=+0.066553282 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106315 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e3f93cf02172a46e97bf96b78a7b SystemUUID:ec21e3f9-3cf0-2172-a46e-97bf96b78a7b BootID:9e798b80-585c-4be9-a285-03490c834408 Filesystems:[{Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-479 DeviceMajor:0 DeviceMinor:479 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52d0b81c86626b6b09dcf955513939173bab68d93cca9207eda1c2d351f3d655/userdata/shm DeviceMajor:0 DeviceMinor:497 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:297 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:342 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:436 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:442 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/676128efd1ec3d1cabda721eefbfe19ede0f6625081a57a8ac54f1482ee85f68/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:445 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/871c9271-d966-4c0d-b367-be51d1d4a79d/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:489 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-533 DeviceMajor:0 DeviceMinor:533 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:272 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:437 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~projected/kube-api-access-hn6dx DeviceMajor:0 DeviceMinor:276 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:301 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:329 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:439 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d86b7d9-2e05-4f95-b002-931249ab66e2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:270 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:443 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ce5ba25c-da20-40d5-b02f-89e94aaad0d2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9e9f60d6c2931c6b6f0f7b6523a3f4e91f39eb6834d1edf73fe15007a760d1a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:340 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:343 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:444 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f2b77242-e22d-4689-9fd6-545ef1da88c4/volumes/kubernetes.io~projected/kube-api-access-9nvxp DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fbcb57cf18754b006c8d806fd1e4164b7234bc19cd69677f3841ceb9bee960e/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d77554d-8094-4546-8fe7-0edf66752a5e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:493 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/433f3b2041f96c84031e68c7d26bdf986bda69de52d983aaa7b3f97bc2f6f05c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~projected/kube-api-access-b5x6c DeviceMajor:0 DeviceMinor:331 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b42b15eb9b8d77cc4bcdc91e1f36a8c3e86bede2d68f98a109038d20725dd9c/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-512 DeviceMajor:0 DeviceMinor:512 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/89a132e7-7020-4c9e-accd-89b0a61d73b0/volumes/kubernetes.io~projected/kube-api-access-5n855 DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:304 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb627f0ff19dec1f579603dfb93d7d049ce7e150e4f7cde6802fa6d4ab735564/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a06c77803ed1a18115d666293b30bb15c5045a523b6b6ccdedae5ae879e71cde/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbbbc1ad-2bb2-469a-8b29-94760bce83cc/volumes/kubernetes.io~projected/kube-api-access-q8hs8 DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~projected/kube-api-access-zqmp8 DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:307 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc878e8c-7351-4591-9939-4d33d63b6d1e/volumes/kubernetes.io~projected/kube-api-access-zxh5h DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/871c9271-d966-4c0d-b367-be51d1d4a79d/volumes/kubernetes.io~projected/kube-api-access-btts9 DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~projected/kube-api-access-5xb5h DeviceMajor:0 DeviceMinor:344 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:371 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0d1b0b56f4aa1b164fe8d1d0676e21f5355ddd50d625d3e9809d384e39bfab0/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volumes/kubernetes.io~projected/kube-api-access-jw9k9 DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-61 DeviceMajor:0 DeviceMinor:61 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:303 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:441 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~projected/kube-api-access-p5lf6 DeviceMajor:0 DeviceMinor:431 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6e21717adb75fa3fdf56f43294b1cb5389c0a5ffa9280c7ce635560fedcb2a8/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:330 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:438 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:440 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27916db1825d09aa9f0d5b6298dbd3b917abac7cfb5f0386047b0cb1cc85d0a9/userdata/shm DeviceMajor:0 DeviceMinor:495 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa09d3e6f1b46069f3cef0db14329bda4be2d2952341a287404ce35edb72f55c/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7acf902a75afca9f4419ca945c7c017a5fe82b3cefb108cab39234025d4c4f25/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:341 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~projected/kube-api-access-wqckm DeviceMajor:0 DeviceMinor:511 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:265 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:302 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:339 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3/volumes/kubernetes.io~projected/kube-api-access-nwdpg DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:303 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:437 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a21962c6-ef9b-413f-abe8-a93835acebbc/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/272dcf18-8186-45d9-9316-8c5a0efc951b/volumes/kubernetes.io~projected/kube-api-access-qhmql DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/353c68dc-e91f-445b-b4f0-386cf1a9c8e7/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a2c398fa990b15181a4e101f345ddf18c076806db928e3f9d01c1dbbb37d7eb/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~projected/kube-api-access-vsfb8 DeviceMajor:0 DeviceMinor:306 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:370 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~projected/kube-api-access-qtcjs DeviceMajor:0 DeviceMinor:230 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:307 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bf20d347dfa51b6449cf1355defaec52e74b130b6bd1a23a5c77e553d5d398f/userdata/shm DeviceMajor:0 DeviceMinor:518 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d86b7d9-2e05-4f95-b002-931249ab66e2/volumes/kubernetes.io~projected/kube-api-access-8d52x DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:309 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d43a365bed980790e8a1fa057ee1e434d810db1209c724fd4f17c7c67e9da874/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:372 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b52d343a-4ee8-474d-b2ab-6fc96edde2fd/volumes/kubernetes.io~projected/kube-api-access-bv2pn DeviceMajor:0 DeviceMinor:494 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-525 DeviceMajor:0 DeviceMinor:525 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:437 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:508 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0e139b70bb14fa05239c9166e60b22218a5eb30760ad8a4d832a8c726af2cf7/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbbbc1ad-2bb2-469a-8b29-94760bce83cc/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:273 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:307 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:437 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/508bd618fe005a261db3945088046ea1b6101b237b62bc1eb1db72e982437d00/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04cdbf66861fa622868c7baf6998d9d45d6466ac731f195cb70655c86f6a3619/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71e42f68fd77314a41cf0580fb1c8ae2e24a30c25032800afd6bb05ad5da2786/userdata/shm DeviceMajor:0 DeviceMinor:448 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:307 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be0307adac5f8682c13f30b6c44ae7df5217989fe5a8d87c1a9ee18ccc4ca2d1/userdata/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/662790f5602c3adfc5b440570e9c38e9e9690b27315e25f6bb06ba30956a47df/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:305 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a5fc82ca-c9cf-43f1-871c-c156ea050ca6/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:333 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c8994e33-9b53-41d7-8fa8-04463a3ae647/volumes/kubernetes.io~projected/kube-api-access-q5zg5 DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e52eca-6d99-4f5c-bbee-9f9559130add/volumes/kubernetes.io~projected/kube-api-access-v2b6n DeviceMajor:0 DeviceMinor:274 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c4196f8503be0d4a88dda1f4100de07bf9130710453e0c72bbd866f520390b0/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:435 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0a7cbc3bb0733cbcba5fe23b17dfa277d8264ee049ecb08f9a7c19f35584a7b/userdata/shm DeviceMajor:0 DeviceMinor:498 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/353c68dc-e91f-445b-b4f0-386cf1a9c8e7/volumes/kubernetes.io~projected/kube-api-access-bcqjj DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfb9e46b67dc8a2b747d12243c10961fa20445e870fcb4da4157f88ccfe476cf/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73bfd18d-99d5-4967-b748-00d94f5dc398/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:271 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:308 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/volumes/kubernetes.io~projected/kube-api-access-wpbdq DeviceMajor:0 DeviceMinor:373 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4020703a95484315726b9502ccaafd08d26dc3e0287b28771549d24120e66621/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:507 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9dc8719-07a1-41fe-834b-66bbd4e9e19b/volumes/kubernetes.io~projected/kube-api-access-chwjc DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:27916db1825d09a MacAddress:a6:e4:61:4f:5e:42 Speed:10000 Mtu:8901} {Name:2bf20d347dfa51b MacAddress:e6:b4:86:89:ea:fe Speed:10000 Mtu:8901} {Name:433f3b2041f96c8 MacAddress:36:4b:1c:e0:86:35 Speed:10000 Mtu:8901} {Name:52d0b81c86626b6 MacAddress:8e:8d:85:f0:8c:b6 Speed:10000 Mtu:8901} {Name:5c4196f8503be0d MacAddress:7a:f9:59:84:fb:16 Speed:10000 Mtu:8901} {Name:676128efd1ec3d1 MacAddress:e2:22:0f:d5:a7:d8 Speed:10000 Mtu:8901} {Name:71e42f68fd77314 MacAddress:6a:74:8d:91:83:26 Speed:10000 Mtu:8901} {Name:7a2c398fa990b15 MacAddress:7e:5a:b8:23:9e:dc Speed:10000 Mtu:8901} {Name:7acf902a75afca9 MacAddress:16:48:1c:c7:bd:5f Speed:10000 Mtu:8901} {Name:7b42b15eb9b8d77 MacAddress:fe:86:5f:ea:d4:26 Speed:10000 Mtu:8901} {Name:bb627f0ff19dec1 MacAddress:f2:25:53:50:21:93 Speed:10000 Mtu:8901} {Name:bfb9e46b67dc8a2 MacAddress:06:55:b9:f6:80:89 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c0:ae:05:3f Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:07:05:98:cb:ee Speed:0 Mtu:8901} {Name:d43a365bed98079 MacAddress:42:04:6c:12:41:b6 Speed:10000 Mtu:8901} {Name:e0d1b0b56f4aa1b MacAddress:f6:c5:c4:15:91:54 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c0:ae:05:3f Speed:0 Mtu:9001} {Name:f0a7cbc3bb0733c MacAddress:62:cd:41:5b:07:eb Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:32:12:fd:d2:97:cd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:9a:69:1d:92:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:36:59.740519 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.740115 5959 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:36:59.740519 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.740191 5959 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:36:59.740710 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.740538 5959 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:36:59.740808 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.740575 5959 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-96.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:36:59.740879 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.740814 5959 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:36:59.740879 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.740828 5959 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:36:59.740879 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.740848 5959 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:36:59.740879 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.740866 5959 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:36:59.741040 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.741028 5959 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:36:59.741241 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.741228 5959 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:36:59.741379 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.741368 5959 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:36:59.741441 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.741418 5959 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:36:59.741441 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.741433 5959 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:36:59.741529 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.741447 5959 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:36:59.741529 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.741460 5959 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:36:59.741529 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.741472 5959 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:36:59.742213 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.742195 5959 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:36:59.742213 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.742211 5959 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:36:59.742867 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.742844 5959 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:36:59.745887 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.745870 5959 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:36:59.745887 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.745886 5959 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:36:59.746020 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746009 5959 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 18:36:59.746232 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746220 5959 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:36:59.746262 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746252 5959 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:36:59.746359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746348 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:36:59.746388 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746369 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:36:59.746388 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746384 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:36:59.746445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746393 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:36:59.746445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746399 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:36:59.746445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746405 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:36:59.746445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746410 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:36:59.746445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746416 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:36:59.746445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746422 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:36:59.746445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746428 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:36:59.746445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746446 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:36:59.746670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746458 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:36:59.746670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746485 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:36:59.746670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.746495 5959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:36:59.750445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.750429 5959 server.go:1263] "Started kubelet" Oct 30 18:36:59.750920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.750489 5959 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:36:59.750920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.750632 5959 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:36:59.750920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.750732 5959 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:36:59.751021 ip-10-0-133-96 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:36:59.751220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.751096 5959 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:36:59.751845 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.751826 5959 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.757683 5959 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.757712 5959 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.757778 5959 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.757766 5959 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:50 +0000 UTC" deadline="2028-07-11 20:04:29.235406534 +0000 UTC" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.757799 5959 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.757800 5959 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14881h27m29.477609323s" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.757791 5959 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:36:59.757905 5959 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-96.us-east-2.compute.internal\" not found" Oct 30 18:36:59.759740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.759343 5959 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:36:59.760265 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.759933 5959 factory.go:55] Registering systemd factory Oct 30 18:36:59.760265 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.759951 5959 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:36:59.760998 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.760967 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bb2bc4b-5c95-469f-bdca-5fe3862f23ad" volumeName="kubernetes.io/projected/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-kube-api-access-hn6dx" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761000 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761013 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ef9c9a-e45e-447f-b68c-56ef8d59ab96" volumeName="kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761024 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="353c68dc-e91f-445b-b4f0-386cf1a9c8e7" volumeName="kubernetes.io/configmap/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761035 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d86b7d9-2e05-4f95-b002-931249ab66e2" volumeName="kubernetes.io/projected/8d86b7d9-2e05-4f95-b002-931249ab66e2-kube-api-access-8d52x" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761045 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761057 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bfd18d-99d5-4967-b748-00d94f5dc398" volumeName="kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-tls" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761069 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761083 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21962c6-ef9b-413f-abe8-a93835acebbc" volumeName="kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-config" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761094 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-grpc-tls" seLinuxMountContext="" Oct 30 18:36:59.761111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761105 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761117 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761130 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc878e8c-7351-4591-9939-4d33d63b6d1e" volumeName="kubernetes.io/projected/cc878e8c-7351-4591-9939-4d33d63b6d1e-kube-api-access-zxh5h" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761145 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e4cb6c-9268-4dda-a3a8-c8bb503942fb" volumeName="kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761159 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="353c68dc-e91f-445b-b4f0-386cf1a9c8e7" volumeName="kubernetes.io/empty-dir/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-data-volume" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761190 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761202 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761214 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bfd18d-99d5-4967-b748-00d94f5dc398" volumeName="kubernetes.io/empty-dir/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-textfile" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761225 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bfd18d-99d5-4967-b748-00d94f5dc398" volumeName="kubernetes.io/projected/73bfd18d-99d5-4967-b748-00d94f5dc398-kube-api-access-zqmp8" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761240 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761251 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761263 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671f39e7-0d53-41ac-af9a-750a01c38f71" volumeName="kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761275 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761287 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761298 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e4cb6c-9268-4dda-a3a8-c8bb503942fb" volumeName="kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-client-ca-bundle" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761314 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbbbc1ad-2bb2-469a-8b29-94760bce83cc" volumeName="kubernetes.io/secret/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-cert" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761327 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272dcf18-8186-45d9-9316-8c5a0efc951b" volumeName="kubernetes.io/configmap/272dcf18-8186-45d9-9316-8c5a0efc951b-serviceca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761343 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761373 5959 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761378 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761409 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761422 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dd549b7-0627-4c6c-9f3e-e04f01607458" volumeName="kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761436 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8994e33-9b53-41d7-8fa8-04463a3ae647" volumeName="kubernetes.io/empty-dir/c8994e33-9b53-41d7-8fa8-04463a3ae647-tmp-dir" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761448 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bfd18d-99d5-4967-b748-00d94f5dc398" volumeName="kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761459 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dd549b7-0627-4c6c-9f3e-e04f01607458" volumeName="kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761470 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761483 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e52eca-6d99-4f5c-bbee-9f9559130add" volumeName="kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761495 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="353c68dc-e91f-445b-b4f0-386cf1a9c8e7" volumeName="kubernetes.io/secret/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761507 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b1378c8-31d1-4011-80ba-f5c0aad8e3d3" volumeName="kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cni-binary-copy" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761519 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b1378c8-31d1-4011-80ba-f5c0aad8e3d3" volumeName="kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-daemon-config" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761530 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761542 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a132e7-7020-4c9e-accd-89b0a61d73b0" volumeName="kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761556 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bb2bc4b-5c95-469f-bdca-5fe3862f23ad" volumeName="kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761568 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/empty-dir/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-tmp" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761581 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671f39e7-0d53-41ac-af9a-750a01c38f71" volumeName="kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761592 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e52eca-6d99-4f5c-bbee-9f9559130add" volumeName="kubernetes.io/projected/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-api-access-v2b6n" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761640 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671f39e7-0d53-41ac-af9a-750a01c38f71" volumeName="kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761654 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761667 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d86b7d9-2e05-4f95-b002-931249ab66e2" volumeName="kubernetes.io/empty-dir/8d86b7d9-2e05-4f95-b002-931249ab66e2-tmp-dir" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761679 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761691 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/configmap/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-metrics-client-ca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761702 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761714 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce5ba25c-da20-40d5-b02f-89e94aaad0d2" volumeName="kubernetes.io/configmap/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-konnectivity-ca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761732 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e4cb6c-9268-4dda-a3a8-c8bb503942fb" volumeName="kubernetes.io/projected/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-kube-api-access-wpbdq" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761743 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ef9c9a-e45e-447f-b68c-56ef8d59ab96" volumeName="kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-tls" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761755 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761768 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bfd18d-99d5-4967-b748-00d94f5dc398" volumeName="kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-metrics-client-ca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761788 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761799 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a132e7-7020-4c9e-accd-89b0a61d73b0" volumeName="kubernetes.io/projected/89a132e7-7020-4c9e-accd-89b0a61d73b0-kube-api-access-5n855" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761802 5959 factory.go:153] Registering CRI-O factory Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761821 5959 factory.go:223] Registration of the crio container factory successfully Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761870 5959 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761885 5959 factory.go:103] Registering Raw factory Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761819 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b77242-e22d-4689-9fd6-545ef1da88c4" volumeName="kubernetes.io/projected/f2b77242-e22d-4689-9fd6-545ef1da88c4-kube-api-access-9nvxp" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761898 5959 manager.go:1199] Started watching for new ooms in manager Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761900 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761913 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dd549b7-0627-4c6c-9f3e-e04f01607458" volumeName="kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761927 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761940 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5fc82ca-c9cf-43f1-871c-c156ea050ca6" volumeName="kubernetes.io/projected/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-kube-api-access-5xb5h" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761956 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="353c68dc-e91f-445b-b4f0-386cf1a9c8e7" volumeName="kubernetes.io/projected/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-api-access-bcqjj" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761968 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761979 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21962c6-ef9b-413f-abe8-a93835acebbc" volumeName="kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-script-lib" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.761990 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21962c6-ef9b-413f-abe8-a93835acebbc" volumeName="kubernetes.io/projected/a21962c6-ef9b-413f-abe8-a93835acebbc-kube-api-access-jw9k9" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762003 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762015 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dd549b7-0627-4c6c-9f3e-e04f01607458" volumeName="kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762029 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e52eca-6d99-4f5c-bbee-9f9559130add" volumeName="kubernetes.io/empty-dir/d8e52eca-6d99-4f5c-bbee-9f9559130add-volume-directive-shadow" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762041 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762052 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8994e33-9b53-41d7-8fa8-04463a3ae647" volumeName="kubernetes.io/projected/c8994e33-9b53-41d7-8fa8-04463a3ae647-kube-api-access-q5zg5" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762064 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b77242-e22d-4689-9fd6-545ef1da88c4" volumeName="kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-tuned" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762078 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272dcf18-8186-45d9-9316-8c5a0efc951b" volumeName="kubernetes.io/projected/272dcf18-8186-45d9-9316-8c5a0efc951b-kube-api-access-qhmql" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762092 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671f39e7-0d53-41ac-af9a-750a01c38f71" volumeName="kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762103 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762114 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21962c6-ef9b-413f-abe8-a93835acebbc" volumeName="kubernetes.io/secret/a21962c6-ef9b-413f-abe8-a93835acebbc-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762126 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9dc8719-07a1-41fe-834b-66bbd4e9e19b" volumeName="kubernetes.io/projected/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-kube-api-access-chwjc" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762140 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dd549b7-0627-4c6c-9f3e-e04f01607458" volumeName="kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762152 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bb2bc4b-5c95-469f-bdca-5fe3862f23ad" volumeName="kubernetes.io/configmap/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-metrics-client-ca" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762164 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b1378c8-31d1-4011-80ba-f5c0aad8e3d3" volumeName="kubernetes.io/projected/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-kube-api-access-nwdpg" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762175 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762187 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762199 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762213 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e52eca-6d99-4f5c-bbee-9f9559130add" volumeName="kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762225 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ef9c9a-e45e-447f-b68c-56ef8d59ab96" volumeName="kubernetes.io/projected/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-kube-api-access-qtcjs" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762236 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762248 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="871c9271-d966-4c0d-b367-be51d1d4a79d" volumeName="kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762259 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e52eca-6d99-4f5c-bbee-9f9559130add" volumeName="kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:36:59.763400 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762271 5959 manager.go:319] Starting recovery of all containers Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.762270 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671f39e7-0d53-41ac-af9a-750a01c38f71" volumeName="kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765081 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671f39e7-0d53-41ac-af9a-750a01c38f71" volumeName="kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765125 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d77554d-8094-4546-8fe7-0edf66752a5e" volumeName="kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765154 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a132e7-7020-4c9e-accd-89b0a61d73b0" volumeName="kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-binary-copy" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765169 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dd549b7-0627-4c6c-9f3e-e04f01607458" volumeName="kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765183 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bb2bc4b-5c95-469f-bdca-5fe3862f23ad" volumeName="kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765205 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e4cb6c-9268-4dda-a3a8-c8bb503942fb" volumeName="kubernetes.io/empty-dir/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-audit-log" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765223 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765244 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765430 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bfd18d-99d5-4967-b748-00d94f5dc398" volumeName="kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765458 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765481 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765508 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a132e7-7020-4c9e-accd-89b0a61d73b0" volumeName="kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765525 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d86b7d9-2e05-4f95-b002-931249ab66e2" volumeName="kubernetes.io/secret/8d86b7d9-2e05-4f95-b002-931249ab66e2-metrics-tls" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765549 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b52d343a-4ee8-474d-b2ab-6fc96edde2fd" volumeName="kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765564 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765582 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e4cb6c-9268-4dda-a3a8-c8bb503942fb" volumeName="kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765618 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ef9c9a-e45e-447f-b68c-56ef8d59ab96" volumeName="kubernetes.io/configmap/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-metrics-client-ca" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765638 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbbbc1ad-2bb2-469a-8b29-94760bce83cc" volumeName="kubernetes.io/projected/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-kube-api-access-q8hs8" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765658 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b77242-e22d-4689-9fd6-545ef1da88c4" volumeName="kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-tmp" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765678 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671f39e7-0d53-41ac-af9a-750a01c38f71" volumeName="kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765695 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce5ba25c-da20-40d5-b02f-89e94aaad0d2" volumeName="kubernetes.io/secret/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-agent-certs" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765719 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e4cb6c-9268-4dda-a3a8-c8bb503942fb" volumeName="kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765790 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e52eca-6d99-4f5c-bbee-9f9559130add" volumeName="kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-metrics-client-ca" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765807 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6ce326-8d12-49b8-90cd-cf42ab792490" volumeName="kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765831 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671f39e7-0d53-41ac-af9a-750a01c38f71" volumeName="kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765848 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21962c6-ef9b-413f-abe8-a93835acebbc" volumeName="kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-env-overrides" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765872 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e4cb6c-9268-4dda-a3a8-c8bb503942fb" volumeName="kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765922 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dd549b7-0627-4c6c-9f3e-e04f01607458" volumeName="kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.765943 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9dc8719-07a1-41fe-834b-66bbd4e9e19b" volumeName="kubernetes.io/configmap/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-iptables-alerter-script" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.766005 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8565b888-5ed6-4d0f-a91c-87659bedfe00" volumeName="kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.766046 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="871c9271-d966-4c0d-b367-be51d1d4a79d" volumeName="kubernetes.io/projected/871c9271-d966-4c0d-b367-be51d1d4a79d-kube-api-access-btts9" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.766075 5959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d86b7d9-2e05-4f95-b002-931249ab66e2" volumeName="kubernetes.io/configmap/8d86b7d9-2e05-4f95-b002-931249ab66e2-config-volume" seLinuxMountContext="" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.766091 5959 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.766102 5959 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:36:59.767100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.766705 5959 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:36:59.770118 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.770077 5959 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 18:36:59.786868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.786806 5959 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:36:59.788848 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.788811 5959 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:36:59.788848 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.788851 5959 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:36:59.789060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.788882 5959 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:36:59.789060 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:36:59.788926 5959 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 18:36:59.795387 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.795357 5959 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:36:59.812289 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.812244 5959 generic.go:356] "Generic (PLEG): container finished" podID="9fb42ec34eaffa661af422bced7d9962" containerID="564ce27d691bee14ba879cf6090dee2d53a65bc9d723196cc0883d692e19ce06" exitCode=0 Oct 30 18:36:59.819271 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.819236 5959 generic.go:356] "Generic (PLEG): container finished" podID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" exitCode=0 Oct 30 18:36:59.825784 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.825745 5959 generic.go:356] "Generic (PLEG): container finished" podID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerID="8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96" exitCode=0 Oct 30 18:36:59.828643 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.828597 5959 generic.go:356] "Generic (PLEG): container finished" podID="73bfd18d-99d5-4967-b748-00d94f5dc398" containerID="36d8d3c07432c46a3e13ca012defd3b9b69d9dff4dde4bf6c7c26cac29d40e31" exitCode=0 Oct 30 18:36:59.838960 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.838920 5959 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="384a47bca7b28e6fcb71a2460e5fb84005c9171bedc6093b6254d7a4051d7fa5" exitCode=0 Oct 30 18:36:59.838960 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.838954 5959 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="8f60dbcfc8429339cbf23e2cbdb66db36d5c0ed6a8e80d5503e0eebee4e3e537" exitCode=0 Oct 30 18:36:59.838960 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.838963 5959 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="e677eee65bd01e2520544795b1f03d4eb516e7426986cb09faf89413103ad4e9" exitCode=0 Oct 30 18:36:59.839211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.838974 5959 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="3dc775cd7937ff2a70d32dd784a30c4cf8657045518262deec8f99a742540594" exitCode=0 Oct 30 18:36:59.839211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.838983 5959 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="66b4d26a86e233b38599f4e6daeca14ad5ce4c84d5d5791ada1df12e84d2bb44" exitCode=0 Oct 30 18:36:59.839211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.838991 5959 generic.go:356] "Generic (PLEG): container finished" podID="89a132e7-7020-4c9e-accd-89b0a61d73b0" containerID="c60d123982db97d47ce8ea9c2613dfcf5f9209eaefaf453d76997f6bf53eef1b" exitCode=0 Oct 30 18:36:59.889005 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:36:59.888977 5959 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 18:36:59.899276 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.899252 5959 manager.go:324] Recovery completed Oct 30 18:36:59.927934 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.927897 5959 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:36:59.927934 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.927919 5959 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:36:59.928150 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.927945 5959 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:36:59.928150 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.928114 5959 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 18:36:59.928150 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.928130 5959 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 18:36:59.928380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.928154 5959 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 18:36:59.928380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.928161 5959 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 18:36:59.928380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.928169 5959 policy_none.go:50] "Start" Oct 30 18:36:59.928380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.928182 5959 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:36:59.928380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.928193 5959 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:36:59.929303 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.929287 5959 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 18:36:59.929378 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.929307 5959 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 18:36:59.929378 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.929317 5959 policy_none.go:44] "Start" Oct 30 18:36:59.934331 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.934298 5959 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:36:59.937518 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.937498 5959 manager.go:341] "Starting Device Plugin manager" Oct 30 18:36:59.938021 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:36:59.937525 5959 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:36:59.938021 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.938020 5959 server.go:87] "Starting device plugin registration server" Oct 30 18:36:59.938304 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.938290 5959 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:36:59.938373 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.938301 5959 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:36:59.938425 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.938396 5959 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:36:59.938525 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.938493 5959 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:36:59.938577 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.938531 5959 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:36:59.938577 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.938536 5959 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:36:59.938740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.938583 5959 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:59.938521883Z","UUID":"a0f73541-3e92-4528-98fc-f02516bb26ef","Handler":null,"Name":"","Endpoint":""} Oct 30 18:36:59.940012 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.939992 5959 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:36:59.940131 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:36:59.940031 5959 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:36:59.940447 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:36:59.940406 5959 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:37:00.039260 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.039218 5959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:37:00.041389 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.041370 5959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:37:00.041469 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.041404 5959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:37:00.041469 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.041415 5959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-96.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:37:00.041565 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.041490 5959 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.052880 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.052855 5959 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.052944 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.052931 5959 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.089132 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.089059 5959 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal"] Oct 30 18:37:00.089755 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.089723 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"9fb42ec34eaffa661af422bced7d9962","Type":"ContainerStarted","Data":"bbd3429f5d3e4bd68c2d71e5093247a6072903538ac00a466e29ebd082cf5f90"} Oct 30 18:37:00.089829 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.089763 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"9fb42ec34eaffa661af422bced7d9962","Type":"ContainerDied","Data":"564ce27d691bee14ba879cf6090dee2d53a65bc9d723196cc0883d692e19ce06"} Oct 30 18:37:00.089829 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.089778 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"9fb42ec34eaffa661af422bced7d9962","Type":"ContainerStarted","Data":"d6e21717adb75fa3fdf56f43294b1cb5389c0a5ffa9280c7ce635560fedcb2a8"} Oct 30 18:37:00.089829 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.089804 5959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.089923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.089866 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"e8517d60336db91a5d4c1afa2b83dc77","Type":"ContainerStarted","Data":"049db31cc86f71141a5c2679dc6a59b6e83bec61d3e2694f37fa5f82f9cbf521"} Oct 30 18:37:00.089923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.089878 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" event={"ID":"e8517d60336db91a5d4c1afa2b83dc77","Type":"ContainerStarted","Data":"be0307adac5f8682c13f30b6c44ae7df5217989fe5a8d87c1a9ee18ccc4ca2d1"} Oct 30 18:37:00.090029 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.090015 5959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.105159 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.105131 5959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:37:00.105235 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:37:00.105189 5959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.105654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.105636 5959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:37:00.105692 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:37:00.105682 5959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.172064 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.171992 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.172064 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.172030 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.172064 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.172048 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8517d60336db91a5d4c1afa2b83dc77-config\") pod \"kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"e8517d60336db91a5d4c1afa2b83dc77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.272730 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.272687 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.272730 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.272729 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.272933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.272748 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8517d60336db91a5d4c1afa2b83dc77-config\") pod \"kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"e8517d60336db91a5d4c1afa2b83dc77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.272933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.272813 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.272933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.272876 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fb42ec34eaffa661af422bced7d9962-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"9fb42ec34eaffa661af422bced7d9962\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.272933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.272907 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8517d60336db91a5d4c1afa2b83dc77-config\") pod \"kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal\" (UID: \"e8517d60336db91a5d4c1afa2b83dc77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.742503 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.742466 5959 apiserver.go:52] "Watching apiserver" Oct 30 18:37:00.754746 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.754715 5959 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:37:00.755522 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.755498 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-qj49w","openshift-ingress-canary/ingress-canary-f92sm","openshift-monitoring/node-exporter-rrqjz","openshift-network-diagnostics/network-check-target-cbh4k","openshift-network-operator/iptables-alerter-fdhrx","openshift-ovn-kubernetes/ovnkube-node-6lwpq","openshift-cluster-node-tuning-operator/tuned-vw4fs","openshift-console/console-668d98c4ff-w5c7v","openshift-dns/dns-default-45mzz","kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal","openshift-console/console-6f5f79687c-qhrj9","openshift-insights/insights-runtime-extractor-jbqcr","openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg","openshift-monitoring/metrics-server-5c7bcfc979-g65nx","openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4","kube-system/konnectivity-agent-g6tw4","openshift-monitoring/thanos-querier-7fd85567c4-mkwgs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj","openshift-dns/node-resolver-msdfn","openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s","openshift-multus/multus-additional-cni-plugins-snt75","openshift-multus/multus-dbklw","kube-system/global-pull-secret-syncer-628cx","openshift-monitoring/alertmanager-main-0","openshift-image-registry/node-ca-slltd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal","openshift-monitoring/prometheus-k8s-0"] Oct 30 18:37:00.765555 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765520 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.765722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765634 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.765722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765676 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89nks\"" Oct 30 18:37:00.765893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765756 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:37:00.765893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765789 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:37:00.765893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765806 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:37:00.765893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765817 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.765893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765862 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.765893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765870 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.765893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765884 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgk4h\"" Oct 30 18:37:00.765893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.765894 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.766258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.766006 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:37:00.766258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.766165 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.766258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.766169 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s4hwp\"" Oct 30 18:37:00.766258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.766171 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:37:00.766258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.766171 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ldjxj\"" Oct 30 18:37:00.766630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.766588 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:37:00.766630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.766592 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:37:00.766749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.766649 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:37:00.774455 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774422 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-config\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.774631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774478 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:37:00.774690 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774663 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-systemd-units\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.774731 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774716 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-slash\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.774765 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774754 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-systemd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.774812 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774784 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.774897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774878 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-node-log\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.774961 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774916 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a21962c6-ef9b-413f-abe8-a93835acebbc-ovn-node-metrics-cert\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.775005 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774958 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-netns\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.775078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.774997 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-ovn\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.775212 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.775179 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-netd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.775512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.775484 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871c9271-d966-4c0d-b367-be51d1d4a79d-metrics-certs\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:37:00.775593 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.775487 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btts9\" (UniqueName: \"kubernetes.io/projected/871c9271-d966-4c0d-b367-be51d1d4a79d-kube-api-access-btts9\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:37:00.775837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.775806 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-bin\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776059 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776036 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-config\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776125 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776060 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a21962c6-ef9b-413f-abe8-a93835acebbc-ovn-node-metrics-cert\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776125 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776065 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776152 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-env-overrides\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776244 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776213 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-script-lib\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776394 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776375 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw9k9\" (UniqueName: \"kubernetes.io/projected/a21962c6-ef9b-413f-abe8-a93835acebbc-kube-api-access-jw9k9\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776500 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776422 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-kubelet\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776530 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-var-lib-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776730 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776710 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-etc-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776787 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776748 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-log-socket\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776935 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776911 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-ovnkube-script-lib\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.776983 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.776925 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.777029 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.777006 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a21962c6-ef9b-413f-abe8-a93835acebbc-env-overrides\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.783599 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.783573 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw9k9\" (UniqueName: \"kubernetes.io/projected/a21962c6-ef9b-413f-abe8-a93835acebbc-kube-api-access-jw9k9\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.783734 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.783639 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btts9\" (UniqueName: \"kubernetes.io/projected/871c9271-d966-4c0d-b367-be51d1d4a79d-kube-api-access-btts9\") pod \"network-metrics-daemon-qj49w\" (UID: \"871c9271-d966-4c0d-b367-be51d1d4a79d\") " pod="openshift-multus/network-metrics-daemon-qj49w" Oct 30 18:37:00.860646 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.860595 5959 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:37:00.864389 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.864362 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:37:00.864646 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.864621 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:37:00.864737 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.864626 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:37:00.864737 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.864693 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:37:00.864737 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.864693 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjz7h\"" Oct 30 18:37:00.864877 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.864859 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:37:00.864914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.864858 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:37:00.864951 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.864940 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:37:00.865075 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865057 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:37:00.865075 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865071 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:37:00.865629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865594 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:37:00.865727 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865596 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:37:00.865785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865725 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-k7fv4\"" Oct 30 18:37:00.865785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865677 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cxpnh\"" Oct 30 18:37:00.865785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865691 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t5wt5\"" Oct 30 18:37:00.865785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865696 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 18:37:00.865785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865740 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:37:00.865785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865766 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qs4k8\"" Oct 30 18:37:00.865785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865592 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 18:37:00.865986 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865782 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 18:37:00.865986 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.865802 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sfbzk\"" Oct 30 18:37:00.866155 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866132 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:37:00.866251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866183 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:37:00.866251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866186 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gxldj\"" Oct 30 18:37:00.866251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866217 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:37:00.866251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866222 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 18:37:00.866251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866241 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8j87p\"" Oct 30 18:37:00.866417 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866303 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4z4br\"" Oct 30 18:37:00.866417 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866366 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:37:00.866417 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866378 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.866417 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866392 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:37:00.866417 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866411 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.866580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866422 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:37:00.866580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866484 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:37:00.866580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866502 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.866580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866522 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:37:00.866580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866539 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.866580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866561 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-slnc4\"" Oct 30 18:37:00.866580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866566 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.866580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866564 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:37:00.866864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866628 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:37:00.866864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866644 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 18:37:00.866864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866652 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:37:00.866864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866648 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5rxk7\"" Oct 30 18:37:00.866864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866696 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:37:00.866864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866768 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.867090 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866905 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 18:37:00.867090 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866917 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:37:00.867090 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.866982 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:37:00.867363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.867350 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:37:00.867423 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.867387 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 18:37:00.867776 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.867744 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 18:37:00.867929 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.867908 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:37:00.867993 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.867940 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 18:37:00.867993 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.867952 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:37:00.868193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.868175 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 18:37:00.868758 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.868738 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:37:00.868849 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.868819 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:37:00.868889 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.868884 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 18:37:00.868889 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.868891 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 18:37:00.868978 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.868924 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.868978 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.868932 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vtmwt\"" Oct 30 18:37:00.868978 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.868940 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:37:00.869107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.869089 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:37:00.869502 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.869469 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lfvmh\"" Oct 30 18:37:00.869502 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.869493 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.869649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.869512 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:37:00.869649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.869526 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xr42l\"" Oct 30 18:37:00.869649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.869526 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.869649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.869542 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:37:00.869649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.869562 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.870082 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870054 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-phzbf\"" Oct 30 18:37:00.870166 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870098 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.870166 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870111 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.870166 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870136 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 18:37:00.870309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870193 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.870309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870200 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 18:37:00.870309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870203 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 18:37:00.870309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870204 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dgrqn\"" Oct 30 18:37:00.870309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870227 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fb5r\"" Oct 30 18:37:00.870309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870254 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 18:37:00.870551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870331 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.870551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870416 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 18:37:00.870551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870419 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8lvuvgi8ronlo\"" Oct 30 18:37:00.870551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870463 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.870551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870466 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:37:00.870551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870488 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kqrcb\"" Oct 30 18:37:00.870551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870499 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 18:37:00.870933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.870915 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4ebv7a0m728dl\"" Oct 30 18:37:00.871414 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.871402 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.871477 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.871451 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8njxf\"" Oct 30 18:37:00.871744 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.871729 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-31139jp48a13i\"" Oct 30 18:37:00.871805 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.871745 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.871805 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.871730 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:37:00.871805 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.871762 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnnzq\"" Oct 30 18:37:00.871919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.871784 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:37:00.872267 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.872254 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:37:00.873037 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.873021 5959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.873141 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.873128 5959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.878591 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.878563 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-systemd\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.878864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.878838 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d86b7d9-2e05-4f95-b002-931249ab66e2-metrics-tls\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:00.878926 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.878888 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.878991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.878924 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.878991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.878957 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.878991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.878987 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-dbus\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:37:00.879119 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879019 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqjj\" (UniqueName: \"kubernetes.io/projected/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-api-access-bcqjj\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.879119 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879049 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-wtmp\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.879119 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879079 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-daemon-config\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.879119 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879108 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-multus-certs\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.879302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879130 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-etc-kubernetes\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.879302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879161 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d52x\" (UniqueName: \"kubernetes.io/projected/8d86b7d9-2e05-4f95-b002-931249ab66e2-kube-api-access-8d52x\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:00.879302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879191 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.879302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879228 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8994e33-9b53-41d7-8fa8-04463a3ae647-hosts-file\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:37:00.879302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879258 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.879302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879283 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-metrics-server-audit-profiles\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.879554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879316 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-var-lib-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.879554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879347 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-log-socket\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.879554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879378 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.879554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879410 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.879554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879433 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-sys\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.879554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879465 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.879554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879495 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.879554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879526 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-tls\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879557 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879581 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-host-slash\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879627 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-cnibin\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879662 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879694 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879734 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879761 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879797 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879826 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-cert\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879871 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nvxp\" (UniqueName: \"kubernetes.io/projected/f2b77242-e22d-4689-9fd6-545ef1da88c4-kube-api-access-9nvxp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.879923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879912 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879956 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.879986 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880023 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880056 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880088 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-metrics-client-ca\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880123 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-audit-log\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880151 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-client-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880194 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysconfig\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880237 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxh5h\" (UniqueName: \"kubernetes.io/projected/cc878e8c-7351-4591-9939-4d33d63b6d1e-kube-api-access-zxh5h\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880279 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:37:00.880372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880322 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-data-volume\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.880801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880356 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cni-binary-copy\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.880801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880426 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-kubelet\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.880801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880474 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-conf-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.880801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.880680 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-client-certs\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.881036 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881010 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-client-certs\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.881107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881082 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.881161 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881108 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d86b7d9-2e05-4f95-b002-931249ab66e2-metrics-tls\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:00.881215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881172 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-var-lib-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.881215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881171 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.881316 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881254 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-audit-log\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.881366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881307 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:37:00.881366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881252 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.881366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881350 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-log-socket\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.881520 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881372 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-daemon-config\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.881520 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881508 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.881650 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881525 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cni-binary-copy\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.881650 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881539 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.881650 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881085 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.881650 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881537 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:37:00.881860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881661 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.881860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881689 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:37:00.881860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881733 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-netd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.881860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881766 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.881860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881830 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.881860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881844 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881869 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881889 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-cert\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881911 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-lib-modules\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881927 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-secret-metrics-server-tls\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881946 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.881985 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-netd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882012 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882032 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-etc-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882078 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-etc-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882099 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-data-volume\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882140 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.882143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882144 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882208 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882229 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882247 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-tuned\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882249 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882346 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-metrics-server-audit-profiles\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882449 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882496 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882510 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882556 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882580 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-client-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.882725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882555 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.883261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882777 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.883261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.882959 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.883261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883045 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d86b7d9-2e05-4f95-b002-931249ab66e2-tmp-dir\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:00.883261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883058 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.883261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883103 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.883261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883117 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-systemd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.883545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883292 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.883545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883314 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d86b7d9-2e05-4f95-b002-931249ab66e2-tmp-dir\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:00.883545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883327 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-node-log\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.883545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883379 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-tuned\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.883545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883445 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-systemd\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.883545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883460 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-openvswitch\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.883545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883470 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-node-log\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.883545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883527 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.883897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883561 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.883897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883595 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.883897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883631 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.883897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883759 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-socket-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.883897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883834 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-iptables-alerter-script\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:37:00.884138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883913 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-ovn\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.884138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.883986 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-run-ovn\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.884138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884022 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:37:00.884138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884038 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.884138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884049 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.884138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884058 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.884138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884128 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwjc\" (UniqueName: \"kubernetes.io/projected/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-kube-api-access-chwjc\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:37:00.884353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884150 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-metrics-client-ca\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.884353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884157 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-iptables-alerter-script\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:37:00.884588 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884568 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:37:00.884737 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884665 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.884737 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884726 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:37:00.884830 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884792 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:37:00.884876 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884845 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.884876 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884855 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.884967 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884902 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.885064 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884970 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.885064 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.884975 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-bin\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.885153 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885061 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.885153 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885123 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b6n\" (UniqueName: \"kubernetes.io/projected/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-api-access-v2b6n\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.885240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885181 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.885240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885181 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.885240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885191 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-cni-bin\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.885240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885213 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-agent-certs\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:37:00.885410 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885250 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-hostroot\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.885410 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885283 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.885410 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885329 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.885410 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885348 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.885410 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885359 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.885410 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885389 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-run\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.885678 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885417 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/272dcf18-8186-45d9-9316-8c5a0efc951b-host\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:37:00.885678 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885524 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.885678 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885572 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-agent-certs\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:37:00.885678 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885631 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.885678 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885660 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.885865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885711 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-systemd-units\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.885865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885730 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.885865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885762 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-systemd-units\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.885865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885770 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-slash\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.885865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885806 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-slash\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.885865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885824 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.886072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885907 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.886072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885921 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.886072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.885969 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.886072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886015 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.886072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886062 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.886260 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886184 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5x6c\" (UniqueName: \"kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.886294 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886261 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-kubernetes\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.886337 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886293 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-conf\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.886337 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886305 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.886337 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886325 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-tmp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.886478 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886372 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:37:00.886478 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886400 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b77242-e22d-4689-9fd6-545ef1da88c4-tmp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.886478 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886409 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.886478 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886458 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.886739 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886583 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:37:00.886739 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886658 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8994e33-9b53-41d7-8fa8-04463a3ae647-tmp-dir\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:37:00.886739 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886714 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:37:00.886875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886779 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-kubelet-config\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:37:00.886875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886786 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8994e33-9b53-41d7-8fa8-04463a3ae647-tmp-dir\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:37:00.886875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886837 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-system-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.886875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886858 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.886875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886864 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-k8s-cni-cncf-io\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.887060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886863 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:37:00.887060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886902 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmql\" (UniqueName: \"kubernetes.io/projected/272dcf18-8186-45d9-9316-8c5a0efc951b-kube-api-access-qhmql\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:37:00.887060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886931 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.887060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886937 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcjs\" (UniqueName: \"kubernetes.io/projected/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-kube-api-access-qtcjs\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:37:00.887060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.886971 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:37:00.887060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887000 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lf6\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.887060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887025 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-metrics-client-ca\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.887060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887051 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-multus\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.887311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887074 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.887311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887106 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86b7d9-2e05-4f95-b002-931249ab66e2-config-volume\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:00.887311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887149 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbdq\" (UniqueName: \"kubernetes.io/projected/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-kube-api-access-wpbdq\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.887311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887175 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n855\" (UniqueName: \"kubernetes.io/projected/89a132e7-7020-4c9e-accd-89b0a61d73b0-kube-api-access-5n855\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.887311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887191 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:37:00.887311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887231 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.887311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887272 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:37:00.887311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887305 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d8e52eca-6d99-4f5c-bbee-9f9559130add-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.887562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887332 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmp8\" (UniqueName: \"kubernetes.io/projected/73bfd18d-99d5-4967-b748-00d94f5dc398-kube-api-access-zqmp8\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.887562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887370 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xb5h\" (UniqueName: \"kubernetes.io/projected/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-kube-api-access-5xb5h\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.887562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887398 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.887562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887409 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86b7d9-2e05-4f95-b002-931249ab66e2-config-volume\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:00.887562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887263 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73bfd18d-99d5-4967-b748-00d94f5dc398-metrics-client-ca\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.887562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887485 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.887562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887523 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.887957 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887677 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d8e52eca-6d99-4f5c-bbee-9f9559130add-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.887957 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887710 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.887957 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887722 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.887957 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887935 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.888139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.887989 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.888139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888039 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zg5\" (UniqueName: \"kubernetes.io/projected/c8994e33-9b53-41d7-8fa8-04463a3ae647-kube-api-access-q5zg5\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:37:00.888139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888048 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89a132e7-7020-4c9e-accd-89b0a61d73b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.888139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888098 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-textfile\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.888346 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888213 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d77554d-8094-4546-8fe7-0edf66752a5e-original-pull-secret\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:37:00.888346 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888288 5959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:37:00.888346 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888290 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-modprobe-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.888475 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888343 5959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:37:00.888475 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888339 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-textfile\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.888475 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:37:00.888424 5959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.888475 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:37:00.888430 5959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-96.us-east-2.compute.internal" Oct 30 18:37:00.888475 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888423 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-var-lib-kubelet\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.888722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888480 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.888722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888551 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-ovn-kubernetes\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.888722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888563 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdpg\" (UniqueName: \"kubernetes.io/projected/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-kube-api-access-nwdpg\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.888722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888631 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-registration-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.888911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888736 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-device-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.888911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888767 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.888911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888835 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.888911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888861 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-grpc-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.889094 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888925 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.889094 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.888977 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/272dcf18-8186-45d9-9316-8c5a0efc951b-serviceca\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:37:00.889196 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889091 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.889196 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889106 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-grpc-tls\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.889196 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889121 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.889196 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889145 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.889196 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889151 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889200 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889202 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-netns\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889235 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889247 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889253 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-run-netns\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889265 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-crio-socket\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889291 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cnibin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889343 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889362 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.889432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889389 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889439 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889423 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889492 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889529 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-root\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889540 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889565 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-tls\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889652 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889687 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-netns\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889722 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-sys\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889700 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/272dcf18-8186-45d9-9316-8c5a0efc951b-serviceca\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889727 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889748 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889799 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889838 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889870 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-tls\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889872 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.889920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889918 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-bin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889950 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889978 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.889992 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890027 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890036 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890059 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-host\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890093 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890094 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-sys-fs\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890145 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890161 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890225 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890241 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890232 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890300 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqckm\" (UniqueName: \"kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890328 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-socket-dir-parent\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890361 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890364 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890386 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-tmp\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890406 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-system-cni-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890426 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hs8\" (UniqueName: \"kubernetes.io/projected/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-kube-api-access-q8hs8\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890472 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890505 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-tmp\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890510 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-konnectivity-ca\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890525 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890599 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890629 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-kubelet\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890656 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890664 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6dx\" (UniqueName: \"kubernetes.io/projected/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-kube-api-access-hn6dx\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890672 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a21962c6-ef9b-413f-abe8-a93835acebbc-host-kubelet\") pod \"ovnkube-node-6lwpq\" (UID: \"a21962c6-ef9b-413f-abe8-a93835acebbc\") " pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890696 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.890703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890704 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ba25c-da20-40d5-b02f-89e94aaad0d2-konnectivity-ca\") pod \"konnectivity-agent-g6tw4\" (UID: \"ce5ba25c-da20-40d5-b02f-89e94aaad0d2\") " pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:37:00.891516 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.890910 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.894914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.894888 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nvxp\" (UniqueName: \"kubernetes.io/projected/f2b77242-e22d-4689-9fd6-545ef1da88c4-kube-api-access-9nvxp\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.894991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.894911 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqjj\" (UniqueName: \"kubernetes.io/projected/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-kube-api-access-bcqjj\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.895143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.895128 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d52x\" (UniqueName: \"kubernetes.io/projected/8d86b7d9-2e05-4f95-b002-931249ab66e2-kube-api-access-8d52x\") pod \"dns-default-45mzz\" (UID: \"8d86b7d9-2e05-4f95-b002-931249ab66e2\") " pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:00.895278 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.895259 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxh5h\" (UniqueName: \"kubernetes.io/projected/cc878e8c-7351-4591-9939-4d33d63b6d1e-kube-api-access-zxh5h\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.895321 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.895263 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2pn\" (UniqueName: \"kubernetes.io/projected/b52d343a-4ee8-474d-b2ab-6fc96edde2fd-kube-api-access-bv2pn\") pod \"network-check-target-cbh4k\" (UID: \"b52d343a-4ee8-474d-b2ab-6fc96edde2fd\") " pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:37:00.895480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.895466 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b6n\" (UniqueName: \"kubernetes.io/projected/d8e52eca-6d99-4f5c-bbee-9f9559130add-kube-api-access-v2b6n\") pod \"kube-state-metrics-5dc8d7d89b-cmsmg\" (UID: \"d8e52eca-6d99-4f5c-bbee-9f9559130add\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cmsmg" Oct 30 18:37:00.896323 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.896302 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwjc\" (UniqueName: \"kubernetes.io/projected/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-kube-api-access-chwjc\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:37:00.896391 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.896325 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmql\" (UniqueName: \"kubernetes.io/projected/272dcf18-8186-45d9-9316-8c5a0efc951b-kube-api-access-qhmql\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:37:00.896536 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.896521 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lf6\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6\") pod \"prometheus-k8s-0\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:00.899141 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.899084 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdpg\" (UniqueName: \"kubernetes.io/projected/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-kube-api-access-nwdpg\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.899663 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.899592 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n855\" (UniqueName: \"kubernetes.io/projected/89a132e7-7020-4c9e-accd-89b0a61d73b0-kube-api-access-5n855\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.899726 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.899714 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbdq\" (UniqueName: \"kubernetes.io/projected/d3e4cb6c-9268-4dda-a3a8-c8bb503942fb-kube-api-access-wpbdq\") pod \"metrics-server-5c7bcfc979-g65nx\" (UID: \"d3e4cb6c-9268-4dda-a3a8-c8bb503942fb\") " pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:00.899798 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.899783 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xb5h\" (UniqueName: \"kubernetes.io/projected/a5fc82ca-c9cf-43f1-871c-c156ea050ca6-kube-api-access-5xb5h\") pod \"thanos-querier-7fd85567c4-mkwgs\" (UID: \"a5fc82ca-c9cf-43f1-871c-c156ea050ca6\") " pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:00.899839 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.899782 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zg5\" (UniqueName: \"kubernetes.io/projected/c8994e33-9b53-41d7-8fa8-04463a3ae647-kube-api-access-q5zg5\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:37:00.900050 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.900029 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8\") pod \"alertmanager-main-0\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:37:00.900080 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.900032 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcjs\" (UniqueName: \"kubernetes.io/projected/e0ef9c9a-e45e-447f-b68c-56ef8d59ab96-kube-api-access-qtcjs\") pod \"prometheus-operator-9cd89b74c-mz6n4\" (UID: \"e0ef9c9a-e45e-447f-b68c-56ef8d59ab96\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mz6n4" Oct 30 18:37:00.900933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.900918 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmp8\" (UniqueName: \"kubernetes.io/projected/73bfd18d-99d5-4967-b748-00d94f5dc398-kube-api-access-zqmp8\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.901673 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.901654 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5x6c\" (UniqueName: \"kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c\") pod \"console-6f5f79687c-qhrj9\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:00.901933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.901914 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hs8\" (UniqueName: \"kubernetes.io/projected/fbbbc1ad-2bb2-469a-8b29-94760bce83cc-kube-api-access-q8hs8\") pod \"ingress-canary-f92sm\" (UID: \"fbbbc1ad-2bb2-469a-8b29-94760bce83cc\") " pod="openshift-ingress-canary/ingress-canary-f92sm" Oct 30 18:37:00.901968 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.901951 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6dx\" (UniqueName: \"kubernetes.io/projected/9bb2bc4b-5c95-469f-bdca-5fe3862f23ad-kube-api-access-hn6dx\") pod \"openshift-state-metrics-7bd7489957-qpm5s\" (UID: \"9bb2bc4b-5c95-469f-bdca-5fe3862f23ad\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpm5s" Oct 30 18:37:00.902120 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.902105 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqckm\" (UniqueName: \"kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm\") pod \"console-668d98c4ff-w5c7v\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:00.991279 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991240 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-multus-certs\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.991279 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991280 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-etc-kubernetes\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991297 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8994e33-9b53-41d7-8fa8-04463a3ae647-hosts-file\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991319 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-sys\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991345 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-host-slash\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991353 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-etc-kubernetes\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991368 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-cnibin\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991383 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-sys\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991393 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991351 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-multus-certs\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991420 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9dc8719-07a1-41fe-834b-66bbd4e9e19b-host-slash\") pod \"iptables-alerter-fdhrx\" (UID: \"d9dc8719-07a1-41fe-834b-66bbd4e9e19b\") " pod="openshift-network-operator/iptables-alerter-fdhrx" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991387 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8994e33-9b53-41d7-8fa8-04463a3ae647-hosts-file\") pod \"node-resolver-msdfn\" (UID: \"c8994e33-9b53-41d7-8fa8-04463a3ae647\") " pod="openshift-dns/node-resolver-msdfn" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991442 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-cnibin\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991456 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991448 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysconfig\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991488 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysconfig\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991490 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-kubelet\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.991517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991515 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-conf-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991541 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-lib-modules\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991542 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-conf-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991568 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-socket-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991520 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-kubelet\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991615 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-hostroot\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991651 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-hostroot\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991657 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-socket-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991653 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-lib-modules\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991652 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-run\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991687 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-run\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991692 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/272dcf18-8186-45d9-9316-8c5a0efc951b-host\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991710 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991714 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/272dcf18-8186-45d9-9316-8c5a0efc951b-host\") pod \"node-ca-slltd\" (UID: \"272dcf18-8186-45d9-9316-8c5a0efc951b\") " pod="openshift-image-registry/node-ca-slltd" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991729 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-kubernetes\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991743 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-conf\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991765 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-kubelet-config\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991780 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-system-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991798 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-k8s-cni-cncf-io\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991804 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-kubernetes\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991825 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-multus\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991842 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-system-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991845 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-k8s-cni-cncf-io\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991866 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-kubelet-config\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991886 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-multus\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991878 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991847 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991891 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-conf\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991955 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-sysctl-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991979 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-modprobe-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.991997 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-var-lib-kubelet\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992012 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-registration-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992026 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-device-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992045 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-crio-socket\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992064 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cnibin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992089 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-var-lib-kubelet\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992096 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-registration-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992091 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-root\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992118 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-device-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992131 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992140 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/353c68dc-e91f-445b-b4f0-386cf1a9c8e7-crio-socket\") pod \"insights-runtime-extractor-jbqcr\" (UID: \"353c68dc-e91f-445b-b4f0-386cf1a9c8e7\") " pod="openshift-insights/insights-runtime-extractor-jbqcr" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992144 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-cnibin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992133 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-modprobe-d\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992157 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-netns\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992161 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-root\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992195 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-run-netns\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992204 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-cni-dir\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992229 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-sys\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992257 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-sys\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992285 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-bin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992314 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-host\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992329 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-sys-fs\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992345 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-socket-dir-parent\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992361 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-system-cni-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992367 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-sys-fs\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.992347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992376 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992396 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-host\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992450 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-systemd\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992470 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-dbus\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992481 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-multus-socket-dir-parent\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992499 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89a132e7-7020-4c9e-accd-89b0a61d73b0-system-cni-dir\") pod \"multus-additional-cni-plugins-snt75\" (UID: \"89a132e7-7020-4c9e-accd-89b0a61d73b0\") " pod="openshift-multus/multus-additional-cni-plugins-snt75" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992515 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b1378c8-31d1-4011-80ba-f5c0aad8e3d3-host-var-lib-cni-bin\") pod \"multus-dbklw\" (UID: \"3b1378c8-31d1-4011-80ba-f5c0aad8e3d3\") " pod="openshift-multus/multus-dbklw" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992511 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-wtmp\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992531 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b77242-e22d-4689-9fd6-545ef1da88c4-etc-systemd\") pod \"tuned-vw4fs\" (UID: \"f2b77242-e22d-4689-9fd6-545ef1da88c4\") " pod="openshift-cluster-node-tuning-operator/tuned-vw4fs" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992556 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d77554d-8094-4546-8fe7-0edf66752a5e-dbus\") pod \"global-pull-secret-syncer-628cx\" (UID: \"7d77554d-8094-4546-8fe7-0edf66752a5e\") " pod="kube-system/global-pull-secret-syncer-628cx" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992592 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73bfd18d-99d5-4967-b748-00d94f5dc398-node-exporter-wtmp\") pod \"node-exporter-rrqjz\" (UID: \"73bfd18d-99d5-4967-b748-00d94f5dc398\") " pod="openshift-monitoring/node-exporter-rrqjz" Oct 30 18:37:00.994938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:00.992528 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc878e8c-7351-4591-9939-4d33d63b6d1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktbj\" (UID: \"cc878e8c-7351-4591-9939-4d33d63b6d1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktbj" Oct 30 18:37:01.035019 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:01.034983 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5f79687c-qhrj9"] Oct 30 18:37:01.265672 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:01.265588 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:01.386322 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:01.386293 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7fd85567c4-mkwgs" Oct 30 18:37:02.966424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:02.966371 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:03.842272 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:03.842243 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-45mzz" Oct 30 18:37:05.203794 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:05.203752 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:05.208323 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:05.208301 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:05.485107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:05.485012 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:09.457159 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:09.457128 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:09.457564 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:09.457354 5959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:37:09.823873 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:09.823788 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:10.113432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:10.113352 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:10.577748 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:10.577705 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:37:19.009141 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:19.009114 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c7bcfc979-g65nx" Oct 30 18:37:20.118046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:20.118009 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:37:23.095627 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:23.095563 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6tw4" Oct 30 18:37:23.341814 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:23.341741 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6lwpq" Oct 30 18:37:26.898411 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:26.898354 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f5f79687c-qhrj9" podUID="8dd549b7-0627-4c6c-9f3e-e04f01607458" containerName="console" containerID="cri-o://c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1" gracePeriod=15 Oct 30 18:37:27.142175 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.142150 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:27.222918 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.222842 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca\") pod \"8dd549b7-0627-4c6c-9f3e-e04f01607458\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " Oct 30 18:37:27.222918 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.222884 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config\") pod \"8dd549b7-0627-4c6c-9f3e-e04f01607458\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " Oct 30 18:37:27.223108 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.222921 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c\" (UniqueName: \"kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c\") pod \"8dd549b7-0627-4c6c-9f3e-e04f01607458\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " Oct 30 18:37:27.223108 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.222962 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp\") pod \"8dd549b7-0627-4c6c-9f3e-e04f01607458\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " Oct 30 18:37:27.223108 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223025 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert\") pod \"8dd549b7-0627-4c6c-9f3e-e04f01607458\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " Oct 30 18:37:27.223108 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223051 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config\") pod \"8dd549b7-0627-4c6c-9f3e-e04f01607458\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " Oct 30 18:37:27.223319 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223105 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert\") pod \"8dd549b7-0627-4c6c-9f3e-e04f01607458\" (UID: \"8dd549b7-0627-4c6c-9f3e-e04f01607458\") " Oct 30 18:37:27.223319 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223278 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp" pod "8dd549b7-0627-4c6c-9f3e-e04f01607458" (UID: "8dd549b7-0627-4c6c-9f3e-e04f01607458"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:27.223429 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223351 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config" pod "8dd549b7-0627-4c6c-9f3e-e04f01607458" (UID: "8dd549b7-0627-4c6c-9f3e-e04f01607458"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:27.223429 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223403 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca" pod "8dd549b7-0627-4c6c-9f3e-e04f01607458" (UID: "8dd549b7-0627-4c6c-9f3e-e04f01607458"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:27.223720 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223688 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert" pod "8dd549b7-0627-4c6c-9f3e-e04f01607458" (UID: "8dd549b7-0627-4c6c-9f3e-e04f01607458"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:27.223990 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223850 5959 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:27.223990 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223864 5959 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:27.223990 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223875 5959 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd549b7-0627-4c6c-9f3e-e04f01607458-tmp\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:27.223990 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.223885 5959 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8dd549b7-0627-4c6c-9f3e-e04f01607458-oauth-serving-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:27.225644 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.225615 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config" pod "8dd549b7-0627-4c6c-9f3e-e04f01607458" (UID: "8dd549b7-0627-4c6c-9f3e-e04f01607458"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:27.225942 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.225912 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert" pod "8dd549b7-0627-4c6c-9f3e-e04f01607458" (UID: "8dd549b7-0627-4c6c-9f3e-e04f01607458"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:27.226032 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.225937 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c" pod "8dd549b7-0627-4c6c-9f3e-e04f01607458" (UID: "8dd549b7-0627-4c6c-9f3e-e04f01607458"). InnerVolumeSpecName "kube-api-access-b5x6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:27.325178 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.325143 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5x6c\" (UniqueName: \"kubernetes.io/projected/8dd549b7-0627-4c6c-9f3e-e04f01607458-kube-api-access-b5x6c\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:27.325178 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.325173 5959 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-serving-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:27.325178 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.325185 5959 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8dd549b7-0627-4c6c-9f3e-e04f01607458-console-oauth-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:27.778281 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:37:27.778247 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:37:27.979741 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.979694 5959 generic.go:356] "Generic (PLEG): container finished" podID="8dd549b7-0627-4c6c-9f3e-e04f01607458" containerID="c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1" exitCode=0 Oct 30 18:37:27.980179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.979766 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5f79687c-qhrj9" Oct 30 18:37:27.980179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.979771 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5f79687c-qhrj9" event={"ID":"8dd549b7-0627-4c6c-9f3e-e04f01607458","Type":"ContainerDied","Data":"c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1"} Oct 30 18:37:27.980179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.979834 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5f79687c-qhrj9" event={"ID":"8dd549b7-0627-4c6c-9f3e-e04f01607458","Type":"ContainerDied","Data":"676128efd1ec3d1cabda721eefbfe19ede0f6625081a57a8ac54f1482ee85f68"} Oct 30 18:37:27.980179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.979854 5959 scope.go:122] "RemoveContainer" containerID="c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1" Oct 30 18:37:27.988125 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.988108 5959 scope.go:122] "RemoveContainer" containerID="c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1" Oct 30 18:37:27.988394 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:37:27.988368 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1\": container with ID starting with c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1 not found: ID does not exist" containerID="c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1" Oct 30 18:37:27.988475 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.988403 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1"} err="failed to get container status \"c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1\": rpc error: code = NotFound desc = could not find container \"c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1\": container with ID starting with c972f57f91e68996d774369d84b7587788f37eb2d69e66d4e011269064bd55f1 not found: ID does not exist" Oct 30 18:37:27.994944 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.994913 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5f79687c-qhrj9"] Oct 30 18:37:27.998787 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:27.998770 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f5f79687c-qhrj9"] Oct 30 18:37:29.795360 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:29.795327 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd549b7-0627-4c6c-9f3e-e04f01607458" path="/var/lib/kubelet/pods/8dd549b7-0627-4c6c-9f3e-e04f01607458/volumes" Oct 30 18:37:30.580762 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:37:30.580730 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cbh4k" Oct 30 18:37:57.779618 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:37:57.779559 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:38:02.983026 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:02.982999 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:02.999393 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:02.999366 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:11.679011 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:11.678973 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:38:11.679482 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:11.679277 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="alertmanager" containerID="cri-o://90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656" gracePeriod=120 Oct 30 18:38:11.679482 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:11.679364 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy-metric" containerID="cri-o://239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5" gracePeriod=120 Oct 30 18:38:11.679482 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:11.679377 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy-web" containerID="cri-o://82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c" gracePeriod=120 Oct 30 18:38:11.679482 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:11.679407 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="prom-label-proxy" containerID="cri-o://ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0" gracePeriod=120 Oct 30 18:38:11.679482 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:11.679436 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy" containerID="cri-o://54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768" gracePeriod=120 Oct 30 18:38:11.679482 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:11.679451 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="config-reloader" containerID="cri-o://09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284" gracePeriod=120 Oct 30 18:38:12.159754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.159722 5959 generic.go:356] "Generic (PLEG): container finished" podID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerID="ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0" exitCode=0 Oct 30 18:38:12.159754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.159748 5959 generic.go:356] "Generic (PLEG): container finished" podID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerID="54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768" exitCode=0 Oct 30 18:38:12.159754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.159755 5959 generic.go:356] "Generic (PLEG): container finished" podID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerID="09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284" exitCode=0 Oct 30 18:38:12.159754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.159760 5959 generic.go:356] "Generic (PLEG): container finished" podID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerID="90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656" exitCode=0 Oct 30 18:38:12.160010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.159797 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerDied","Data":"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0"} Oct 30 18:38:12.160010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.159855 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerDied","Data":"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768"} Oct 30 18:38:12.160010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.159880 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerDied","Data":"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284"} Oct 30 18:38:12.160010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.159902 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerDied","Data":"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656"} Oct 30 18:38:12.945252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:12.945227 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.034546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034462 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034563 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034624 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034823 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034715 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034823 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034762 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034823 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034802 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034976 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034838 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034976 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034889 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034976 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034926 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.034976 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034961 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.035183 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.034992 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.035183 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.035024 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:13.035183 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.035038 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.035183 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.035074 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8\") pod \"4d6ce326-8d12-49b8-90cd-cf42ab792490\" (UID: \"4d6ce326-8d12-49b8-90cd-cf42ab792490\") " Oct 30 18:38:13.035538 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.035515 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-metrics-client-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.035733 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.035538 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:13.035733 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.035625 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.038549 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.038518 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:13.038943 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.038901 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.039077 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.039035 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.039162 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.039135 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.039885 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.039852 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "kube-api-access-vsfb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:13.039885 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.039868 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.039996 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.039908 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.040437 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.040415 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.043820 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.043792 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.050373 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.050347 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config" pod "4d6ce326-8d12-49b8-90cd-cf42ab792490" (UID: "4d6ce326-8d12-49b8-90cd-cf42ab792490"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.136727 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136673 5959 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-main-db\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.136727 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136718 5959 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-main-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.136727 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136730 5959 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-tls-assets\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.136727 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136741 5959 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.137006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136750 5959 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d6ce326-8d12-49b8-90cd-cf42ab792490-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.137006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136760 5959 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-web-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.137006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136769 5959 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.137006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136777 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/4d6ce326-8d12-49b8-90cd-cf42ab792490-kube-api-access-vsfb8\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.137006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136786 5959 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.137006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136794 5959 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d6ce326-8d12-49b8-90cd-cf42ab792490-config-out\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.137006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136803 5959 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-cluster-tls-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.137006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.136812 5959 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d6ce326-8d12-49b8-90cd-cf42ab792490-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:13.166637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.166585 5959 generic.go:356] "Generic (PLEG): container finished" podID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerID="239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5" exitCode=0 Oct 30 18:38:13.166637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.166632 5959 generic.go:356] "Generic (PLEG): container finished" podID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerID="82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c" exitCode=0 Oct 30 18:38:13.166818 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.166677 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerDied","Data":"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5"} Oct 30 18:38:13.166818 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.166700 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.166818 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.166749 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerDied","Data":"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c"} Oct 30 18:38:13.166818 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.166776 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d6ce326-8d12-49b8-90cd-cf42ab792490","Type":"ContainerDied","Data":"d43a365bed980790e8a1fa057ee1e434d810db1209c724fd4f17c7c67e9da874"} Oct 30 18:38:13.166818 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.166808 5959 scope.go:122] "RemoveContainer" containerID="ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0" Oct 30 18:38:13.174846 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.174829 5959 scope.go:122] "RemoveContainer" containerID="239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5" Oct 30 18:38:13.182565 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.182547 5959 scope.go:122] "RemoveContainer" containerID="54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768" Oct 30 18:38:13.189778 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.189759 5959 scope.go:122] "RemoveContainer" containerID="82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c" Oct 30 18:38:13.191924 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.191892 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:38:13.198504 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.198482 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:38:13.198597 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.198585 5959 scope.go:122] "RemoveContainer" containerID="09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284" Oct 30 18:38:13.206062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.206042 5959 scope.go:122] "RemoveContainer" containerID="90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656" Oct 30 18:38:13.213582 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.213562 5959 scope.go:122] "RemoveContainer" containerID="8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96" Oct 30 18:38:13.221709 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.221688 5959 scope.go:122] "RemoveContainer" containerID="ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0" Oct 30 18:38:13.222024 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:13.222002 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0\": container with ID starting with ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0 not found: ID does not exist" containerID="ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0" Oct 30 18:38:13.222097 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.222033 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0"} err="failed to get container status \"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0\": rpc error: code = NotFound desc = could not find container \"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0\": container with ID starting with ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0 not found: ID does not exist" Oct 30 18:38:13.222097 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.222051 5959 scope.go:122] "RemoveContainer" containerID="239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5" Oct 30 18:38:13.222277 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:13.222260 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5\": container with ID starting with 239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5 not found: ID does not exist" containerID="239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5" Oct 30 18:38:13.222312 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.222284 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5"} err="failed to get container status \"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5\": rpc error: code = NotFound desc = could not find container \"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5\": container with ID starting with 239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5 not found: ID does not exist" Oct 30 18:38:13.222312 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.222302 5959 scope.go:122] "RemoveContainer" containerID="54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768" Oct 30 18:38:13.222523 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:13.222508 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768\": container with ID starting with 54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768 not found: ID does not exist" containerID="54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768" Oct 30 18:38:13.222562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.222527 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768"} err="failed to get container status \"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768\": rpc error: code = NotFound desc = could not find container \"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768\": container with ID starting with 54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768 not found: ID does not exist" Oct 30 18:38:13.222562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.222540 5959 scope.go:122] "RemoveContainer" containerID="82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c" Oct 30 18:38:13.222910 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:13.222892 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c\": container with ID starting with 82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c not found: ID does not exist" containerID="82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c" Oct 30 18:38:13.222999 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.222918 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c"} err="failed to get container status \"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c\": rpc error: code = NotFound desc = could not find container \"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c\": container with ID starting with 82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c not found: ID does not exist" Oct 30 18:38:13.222999 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.222937 5959 scope.go:122] "RemoveContainer" containerID="09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284" Oct 30 18:38:13.223234 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:13.223218 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284\": container with ID starting with 09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284 not found: ID does not exist" containerID="09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284" Oct 30 18:38:13.223268 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.223239 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284"} err="failed to get container status \"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284\": rpc error: code = NotFound desc = could not find container \"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284\": container with ID starting with 09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284 not found: ID does not exist" Oct 30 18:38:13.223268 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.223253 5959 scope.go:122] "RemoveContainer" containerID="90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656" Oct 30 18:38:13.223514 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:13.223499 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656\": container with ID starting with 90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656 not found: ID does not exist" containerID="90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656" Oct 30 18:38:13.223561 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.223518 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656"} err="failed to get container status \"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656\": rpc error: code = NotFound desc = could not find container \"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656\": container with ID starting with 90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656 not found: ID does not exist" Oct 30 18:38:13.223561 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.223531 5959 scope.go:122] "RemoveContainer" containerID="8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96" Oct 30 18:38:13.223781 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:13.223765 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96\": container with ID starting with 8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96 not found: ID does not exist" containerID="8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96" Oct 30 18:38:13.223854 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.223790 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96"} err="failed to get container status \"8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96\": rpc error: code = NotFound desc = could not find container \"8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96\": container with ID starting with 8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96 not found: ID does not exist" Oct 30 18:38:13.223854 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.223808 5959 scope.go:122] "RemoveContainer" containerID="ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0" Oct 30 18:38:13.224451 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.224433 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0"} err="failed to get container status \"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0\": rpc error: code = NotFound desc = could not find container \"ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0\": container with ID starting with ee288a9ea398ceeac2f2da0c698b074a1e76e29960a6c0c659312742cd9958f0 not found: ID does not exist" Oct 30 18:38:13.224496 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.224452 5959 scope.go:122] "RemoveContainer" containerID="239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5" Oct 30 18:38:13.224686 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.224668 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5"} err="failed to get container status \"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5\": rpc error: code = NotFound desc = could not find container \"239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5\": container with ID starting with 239fe82273b98255ddd9e955d628b68c40910328842909b8ac74beef6c7f39a5 not found: ID does not exist" Oct 30 18:38:13.224744 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.224686 5959 scope.go:122] "RemoveContainer" containerID="54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768" Oct 30 18:38:13.224917 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.224898 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768"} err="failed to get container status \"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768\": rpc error: code = NotFound desc = could not find container \"54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768\": container with ID starting with 54614e45af659f9f00940a7ee89e3b0d411da8b5673f12bbe7fe809d68bd7768 not found: ID does not exist" Oct 30 18:38:13.224981 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.224919 5959 scope.go:122] "RemoveContainer" containerID="82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c" Oct 30 18:38:13.225160 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.225139 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c"} err="failed to get container status \"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c\": rpc error: code = NotFound desc = could not find container \"82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c\": container with ID starting with 82195209f067fca8e7305e1dd70b11bda2453c6103577e85fafa002f85f7923c not found: ID does not exist" Oct 30 18:38:13.225208 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.225161 5959 scope.go:122] "RemoveContainer" containerID="09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284" Oct 30 18:38:13.225356 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.225339 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284"} err="failed to get container status \"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284\": rpc error: code = NotFound desc = could not find container \"09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284\": container with ID starting with 09545fb12c62003effc4016bb174d56c79dc4fecf66574222c1d46ab29954284 not found: ID does not exist" Oct 30 18:38:13.225428 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.225358 5959 scope.go:122] "RemoveContainer" containerID="90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656" Oct 30 18:38:13.225707 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.225599 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656"} err="failed to get container status \"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656\": rpc error: code = NotFound desc = could not find container \"90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656\": container with ID starting with 90a3904e0d66d3399832ee37ed047d78acf4199f0c39417c35f867a44e1af656 not found: ID does not exist" Oct 30 18:38:13.225759 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.225712 5959 scope.go:122] "RemoveContainer" containerID="8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96" Oct 30 18:38:13.225942 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.225924 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96"} err="failed to get container status \"8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96\": rpc error: code = NotFound desc = could not find container \"8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96\": container with ID starting with 8927590d89b9110cfb057af20e77b64319f29085f7632255f6ba79f330c59b96 not found: ID does not exist" Oct 30 18:38:13.240812 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.240778 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:38:13.241198 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241173 5959 cpu_manager.go:414] "containerMap: removing container" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy-web" Oct 30 18:38:13.241198 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241194 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy-web" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241212 5959 cpu_manager.go:414] "containerMap: removing container" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="init-config-reloader" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241218 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="init-config-reloader" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241226 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8dd549b7-0627-4c6c-9f3e-e04f01607458" containerName="console" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241231 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd549b7-0627-4c6c-9f3e-e04f01607458" containerName="console" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241243 5959 cpu_manager.go:414] "containerMap: removing container" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="prom-label-proxy" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241248 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="prom-label-proxy" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241255 5959 cpu_manager.go:414] "containerMap: removing container" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy-metric" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241260 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy-metric" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241268 5959 cpu_manager.go:414] "containerMap: removing container" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241273 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241279 5959 cpu_manager.go:414] "containerMap: removing container" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="alertmanager" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241284 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="alertmanager" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241291 5959 cpu_manager.go:414] "containerMap: removing container" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="config-reloader" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241296 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="config-reloader" Oct 30 18:38:13.241342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241348 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy-web" Oct 30 18:38:13.241843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241356 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="init-config-reloader" Oct 30 18:38:13.241843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241362 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd549b7-0627-4c6c-9f3e-e04f01607458" containerName="console" Oct 30 18:38:13.241843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241369 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="prom-label-proxy" Oct 30 18:38:13.241843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241374 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy-metric" Oct 30 18:38:13.241843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241380 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="kube-rbac-proxy" Oct 30 18:38:13.241843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241384 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="alertmanager" Oct 30 18:38:13.241843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.241389 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" containerName="config-reloader" Oct 30 18:38:13.244844 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.244827 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.251969 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.251950 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:38:13.252054 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.251959 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:38:13.252054 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.251998 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:38:13.252156 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.252135 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:38:13.252463 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.252448 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vtmwt\"" Oct 30 18:38:13.254643 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.254626 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:38:13.254784 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.254769 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:38:13.254837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.254790 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:38:13.254837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.254776 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:38:13.260652 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.260588 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:38:13.266325 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.266296 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:38:13.337983 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.337881 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.337983 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.337921 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4hr\" (UniqueName: \"kubernetes.io/projected/641f46c9-7931-4981-89ab-3f056fed93e1-kube-api-access-kf4hr\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.337983 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.337944 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338002 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/641f46c9-7931-4981-89ab-3f056fed93e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338160 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/641f46c9-7931-4981-89ab-3f056fed93e1-config-out\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338228 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338264 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/641f46c9-7931-4981-89ab-3f056fed93e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338416 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338336 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641f46c9-7931-4981-89ab-3f056fed93e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338416 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338368 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338416 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338394 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338416 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338414 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/641f46c9-7931-4981-89ab-3f056fed93e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338534 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338445 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-web-config\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.338534 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.338515 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439031 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.438994 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439031 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439032 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/641f46c9-7931-4981-89ab-3f056fed93e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439051 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641f46c9-7931-4981-89ab-3f056fed93e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439086 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439121 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439149 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/641f46c9-7931-4981-89ab-3f056fed93e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439169 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-web-config\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439196 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439230 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439263 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4hr\" (UniqueName: \"kubernetes.io/projected/641f46c9-7931-4981-89ab-3f056fed93e1-kube-api-access-kf4hr\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439309 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439334 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/641f46c9-7931-4981-89ab-3f056fed93e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439366 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/641f46c9-7931-4981-89ab-3f056fed93e1-config-out\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439492 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/641f46c9-7931-4981-89ab-3f056fed93e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.439659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.439542 5959 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:38:13.440308 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.440103 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641f46c9-7931-4981-89ab-3f056fed93e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.440911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.440889 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/641f46c9-7931-4981-89ab-3f056fed93e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.442783 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.442703 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.442783 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.442739 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.443435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.443070 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.443435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.443110 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.443435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.443125 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/641f46c9-7931-4981-89ab-3f056fed93e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.443435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.443181 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.443435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.443397 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-web-config\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.443877 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.443860 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/641f46c9-7931-4981-89ab-3f056fed93e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.444154 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.444136 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/641f46c9-7931-4981-89ab-3f056fed93e1-config-out\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.450396 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.450375 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4hr\" (UniqueName: \"kubernetes.io/projected/641f46c9-7931-4981-89ab-3f056fed93e1-kube-api-access-kf4hr\") pod \"alertmanager-main-0\" (UID: \"641f46c9-7931-4981-89ab-3f056fed93e1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.570492 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.570454 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:38:13.794145 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.794110 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6ce326-8d12-49b8-90cd-cf42ab792490" path="/var/lib/kubelet/pods/4d6ce326-8d12-49b8-90cd-cf42ab792490/volumes" Oct 30 18:38:13.853181 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:13.853151 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:38:13.856703 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:38:13.856677 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641f46c9_7931_4981_89ab_3f056fed93e1.slice/crio-ded0e7d99efbca852e0d28762d1ab3020842448f60c7c8294ca71cef7427f77c WatchSource:0}: Error finding container ded0e7d99efbca852e0d28762d1ab3020842448f60c7c8294ca71cef7427f77c: Status 404 returned error can't find the container with id ded0e7d99efbca852e0d28762d1ab3020842448f60c7c8294ca71cef7427f77c Oct 30 18:38:14.172042 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:14.172011 5959 generic.go:356] "Generic (PLEG): container finished" podID="641f46c9-7931-4981-89ab-3f056fed93e1" containerID="6f97204d9d5b647d29a97f55a2f2972b43f8bb69de5b04a6abca1fea76f7af69" exitCode=0 Oct 30 18:38:14.172499 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:14.172107 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"641f46c9-7931-4981-89ab-3f056fed93e1","Type":"ContainerDied","Data":"6f97204d9d5b647d29a97f55a2f2972b43f8bb69de5b04a6abca1fea76f7af69"} Oct 30 18:38:14.172499 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:14.172213 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"641f46c9-7931-4981-89ab-3f056fed93e1","Type":"ContainerStarted","Data":"ded0e7d99efbca852e0d28762d1ab3020842448f60c7c8294ca71cef7427f77c"} Oct 30 18:38:15.180101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.180063 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"641f46c9-7931-4981-89ab-3f056fed93e1","Type":"ContainerStarted","Data":"958b415da598049d7df016e67d319ea03f874a0b61da7dcc3e790c351080de8a"} Oct 30 18:38:15.180472 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.180145 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"641f46c9-7931-4981-89ab-3f056fed93e1","Type":"ContainerStarted","Data":"9bffd06d04995139fb4fa02b689d695f287fa1475a9eb812b691c47bdf70cac4"} Oct 30 18:38:15.180472 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.180185 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"641f46c9-7931-4981-89ab-3f056fed93e1","Type":"ContainerStarted","Data":"b0bac73defe83cc3f43590a4c1dd02576f107042bea10aa2e6ca1c76cac2ec0a"} Oct 30 18:38:15.180472 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.180222 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"641f46c9-7931-4981-89ab-3f056fed93e1","Type":"ContainerStarted","Data":"fd5875bae5b870fa485d74d3c9ecb4853d57351427934c3b5ecaeed280d194b7"} Oct 30 18:38:15.180472 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.180262 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"641f46c9-7931-4981-89ab-3f056fed93e1","Type":"ContainerStarted","Data":"e203fb7e31718c156b364223ea07623c8bb221645f4bc8da8518eb9df10391c7"} Oct 30 18:38:15.180472 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.180296 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"641f46c9-7931-4981-89ab-3f056fed93e1","Type":"ContainerStarted","Data":"a4985c9d8417ab764ca85504093ef17adc1baf5d9c1905a2f885fddcc76291be"} Oct 30 18:38:15.211583 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.211496 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.200331351 podStartE2EDuration="2.211477581s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.14623ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:15.210096995 +0000 UTC m=+75.538508391" watchObservedRunningTime="2026-10-30 18:38:15.211477581 +0000 UTC m=+75.539888967" Oct 30 18:38:15.718936 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.718904 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5784486569-4gvpd"] Oct 30 18:38:15.722925 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.722901 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.728655 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.728625 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 18:38:15.732131 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.731806 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 18:38:15.741125 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.741099 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5784486569-4gvpd"] Oct 30 18:38:15.763458 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.763427 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d5c\" (UniqueName: \"kubernetes.io/projected/d144b776-a2c9-450c-b189-e299d6d447f6-kube-api-access-f9d5c\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.763458 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.763465 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-metrics-client-ca\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.763695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.763483 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.763695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.763641 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-secret-telemeter-client\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.763695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.763673 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-telemeter-client-tls\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.763826 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.763693 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-serving-certs-ca-bundle\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.763826 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.763719 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.763900 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.763877 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-federate-client-tls\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.793726 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.793692 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 18:38:15.793945 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.793922 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-v2x2s\"" Oct 30 18:38:15.794014 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.793756 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 18:38:15.794014 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.793740 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 18:38:15.794419 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.794402 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 18:38:15.865119 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865083 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-serving-certs-ca-bundle\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.865119 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865119 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.865359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865169 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-federate-client-tls\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.865359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865193 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d5c\" (UniqueName: \"kubernetes.io/projected/d144b776-a2c9-450c-b189-e299d6d447f6-kube-api-access-f9d5c\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.865359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865244 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-metrics-client-ca\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.865359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865274 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.865359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865340 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-secret-telemeter-client\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.865583 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865388 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-telemeter-client-tls\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.866008 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.865980 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-serving-certs-ca-bundle\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.866255 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.866231 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-metrics-client-ca\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.866454 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.866434 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d144b776-a2c9-450c-b189-e299d6d447f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.868289 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.868261 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.868568 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.868547 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-federate-client-tls\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.868568 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.868560 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-telemeter-client-tls\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.868695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.868569 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d144b776-a2c9-450c-b189-e299d6d447f6-secret-telemeter-client\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:15.873662 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:15.873642 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d5c\" (UniqueName: \"kubernetes.io/projected/d144b776-a2c9-450c-b189-e299d6d447f6-kube-api-access-f9d5c\") pod \"telemeter-client-5784486569-4gvpd\" (UID: \"d144b776-a2c9-450c-b189-e299d6d447f6\") " pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:16.043337 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.043240 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" Oct 30 18:38:16.139363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.137854 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:38:16.139363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.138214 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="prometheus" containerID="cri-o://43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" gracePeriod=600 Oct 30 18:38:16.139363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.138633 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" gracePeriod=600 Oct 30 18:38:16.139363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.138741 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy" containerID="cri-o://9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" gracePeriod=600 Oct 30 18:38:16.139363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.138819 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy-web" containerID="cri-o://f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" gracePeriod=600 Oct 30 18:38:16.139363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.138909 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="thanos-sidecar" containerID="cri-o://d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" gracePeriod=600 Oct 30 18:38:16.139363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.138992 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="config-reloader" containerID="cri-o://71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" gracePeriod=600 Oct 30 18:38:16.330924 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.330887 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5784486569-4gvpd"] Oct 30 18:38:16.333716 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:38:16.333689 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd144b776_a2c9_450c_b189_e299d6d447f6.slice/crio-81389ff25992e917e2f5ed08935ee98df1e619ecc710f78522598af5b2ffbbbb WatchSource:0}: Error finding container 81389ff25992e917e2f5ed08935ee98df1e619ecc710f78522598af5b2ffbbbb: Status 404 returned error can't find the container with id 81389ff25992e917e2f5ed08935ee98df1e619ecc710f78522598af5b2ffbbbb Oct 30 18:38:16.335779 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.335761 5959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:38:16.449944 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.449916 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:16.474096 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474042 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474288 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474115 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474288 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474153 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474288 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474182 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474288 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474214 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474288 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474245 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474288 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474278 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474315 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474353 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474385 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474425 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474459 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474498 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474527 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474550 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474579 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474630 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474660 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.474909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.474693 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.475207 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.475082 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:16.477579 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.475737 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:16.477579 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.475794 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db\") pod \"8565b888-5ed6-4d0f-a91c-87659bedfe00\" (UID: \"8565b888-5ed6-4d0f-a91c-87659bedfe00\") " Oct 30 18:38:16.477579 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.476091 5959 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-metrics-client-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.477579 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.476113 5959 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.477579 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.476384 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:16.477579 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.476464 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:16.478480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.477906 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:16.478480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.477975 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:16.478480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.478011 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:16.479747 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.479107 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:16.479747 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.479126 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.480107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.480021 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.480107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.480042 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:16.480107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.480072 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "kube-api-access-p5lf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:16.480370 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.480180 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.480637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.480594 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.480834 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.480805 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.481236 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.481212 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.481354 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.481308 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:16.481429 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.481406 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.481560 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.481538 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.497002 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.496963 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config" pod "8565b888-5ed6-4d0f-a91c-87659bedfe00" (UID: "8565b888-5ed6-4d0f-a91c-87659bedfe00"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:16.577551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577467 5959 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577498 5959 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577508 5959 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-metrics-client-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577518 5959 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577529 5959 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-kube-rbac-proxy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577538 5959 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577548 5959 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-web-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577557 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5lf6\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-kube-api-access-p5lf6\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577566 5959 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577575 5959 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577583 5959 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577593 5959 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8565b888-5ed6-4d0f-a91c-87659bedfe00-tls-assets\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577628 5959 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577637 5959 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577645 5959 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-secret-grpc-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577654 5959 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8565b888-5ed6-4d0f-a91c-87659bedfe00-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577662 5959 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-config-out\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:16.577919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:16.577670 5959 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8565b888-5ed6-4d0f-a91c-87659bedfe00-prometheus-k8s-db\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:17.204223 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204185 5959 generic.go:356] "Generic (PLEG): container finished" podID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" exitCode=0 Oct 30 18:38:17.204223 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204221 5959 generic.go:356] "Generic (PLEG): container finished" podID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" exitCode=0 Oct 30 18:38:17.204435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204232 5959 generic.go:356] "Generic (PLEG): container finished" podID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" exitCode=0 Oct 30 18:38:17.204435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204244 5959 generic.go:356] "Generic (PLEG): container finished" podID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" exitCode=0 Oct 30 18:38:17.204435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204252 5959 generic.go:356] "Generic (PLEG): container finished" podID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" exitCode=0 Oct 30 18:38:17.204435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204260 5959 generic.go:356] "Generic (PLEG): container finished" podID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" exitCode=0 Oct 30 18:38:17.204435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204273 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerDied","Data":"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626"} Oct 30 18:38:17.204435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204352 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.204435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204384 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerDied","Data":"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0"} Oct 30 18:38:17.204435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204426 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerDied","Data":"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c"} Oct 30 18:38:17.204784 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204453 5959 scope.go:122] "RemoveContainer" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" Oct 30 18:38:17.204784 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204469 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerDied","Data":"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0"} Oct 30 18:38:17.204784 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204512 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerDied","Data":"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2"} Oct 30 18:38:17.204784 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204550 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerDied","Data":"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d"} Oct 30 18:38:17.204784 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.204590 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8565b888-5ed6-4d0f-a91c-87659bedfe00","Type":"ContainerDied","Data":"71e42f68fd77314a41cf0580fb1c8ae2e24a30c25032800afd6bb05ad5da2786"} Oct 30 18:38:17.206079 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.206008 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" event={"ID":"d144b776-a2c9-450c-b189-e299d6d447f6","Type":"ContainerStarted","Data":"81389ff25992e917e2f5ed08935ee98df1e619ecc710f78522598af5b2ffbbbb"} Oct 30 18:38:17.214231 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.214201 5959 scope.go:122] "RemoveContainer" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" Oct 30 18:38:17.224481 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.224462 5959 scope.go:122] "RemoveContainer" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" Oct 30 18:38:17.233314 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.233292 5959 scope.go:122] "RemoveContainer" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" Oct 30 18:38:17.234091 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.234066 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:38:17.241129 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.241104 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:38:17.244637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.244619 5959 scope.go:122] "RemoveContainer" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" Oct 30 18:38:17.253424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.253404 5959 scope.go:122] "RemoveContainer" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" Oct 30 18:38:17.262922 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.262891 5959 scope.go:122] "RemoveContainer" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" Oct 30 18:38:17.273854 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.273832 5959 scope.go:122] "RemoveContainer" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" Oct 30 18:38:17.274146 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274128 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:38:17.274210 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:17.274191 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": container with ID starting with 8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626 not found: ID does not exist" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" Oct 30 18:38:17.274256 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274221 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626"} err="failed to get container status \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": rpc error: code = NotFound desc = could not find container \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": container with ID starting with 8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626 not found: ID does not exist" Oct 30 18:38:17.274256 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274240 5959 scope.go:122] "RemoveContainer" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" Oct 30 18:38:17.274508 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274493 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy-web" Oct 30 18:38:17.274554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274511 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy-web" Oct 30 18:38:17.274554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274524 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="thanos-sidecar" Oct 30 18:38:17.274554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274529 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="thanos-sidecar" Oct 30 18:38:17.274554 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:17.274533 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": container with ID starting with 9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0 not found: ID does not exist" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" Oct 30 18:38:17.274554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274545 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="init-config-reloader" Oct 30 18:38:17.274554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274551 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="init-config-reloader" Oct 30 18:38:17.274554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274549 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0"} err="failed to get container status \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": rpc error: code = NotFound desc = could not find container \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": container with ID starting with 9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0 not found: ID does not exist" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274562 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy-thanos" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274562 5959 scope.go:122] "RemoveContainer" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274567 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy-thanos" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274573 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274578 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274589 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="config-reloader" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274593 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="config-reloader" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274624 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="prometheus" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274632 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="prometheus" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274709 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="prometheus" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274725 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy-web" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274735 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="thanos-sidecar" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274746 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="init-config-reloader" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274755 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy-thanos" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274764 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="kube-rbac-proxy" Oct 30 18:38:17.274811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274773 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" containerName="config-reloader" Oct 30 18:38:17.275512 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:17.274855 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": container with ID starting with f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c not found: ID does not exist" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" Oct 30 18:38:17.275512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274886 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c"} err="failed to get container status \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": rpc error: code = NotFound desc = could not find container \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": container with ID starting with f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c not found: ID does not exist" Oct 30 18:38:17.275512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.274918 5959 scope.go:122] "RemoveContainer" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" Oct 30 18:38:17.275512 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:17.275352 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": container with ID starting with d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0 not found: ID does not exist" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" Oct 30 18:38:17.275512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.275385 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0"} err="failed to get container status \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": rpc error: code = NotFound desc = could not find container \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": container with ID starting with d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0 not found: ID does not exist" Oct 30 18:38:17.275512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.275410 5959 scope.go:122] "RemoveContainer" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" Oct 30 18:38:17.275860 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:17.275838 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": container with ID starting with 71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2 not found: ID does not exist" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" Oct 30 18:38:17.275913 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.275868 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2"} err="failed to get container status \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": rpc error: code = NotFound desc = could not find container \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": container with ID starting with 71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2 not found: ID does not exist" Oct 30 18:38:17.275913 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.275882 5959 scope.go:122] "RemoveContainer" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" Oct 30 18:38:17.276265 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:17.276231 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": container with ID starting with 43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d not found: ID does not exist" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" Oct 30 18:38:17.276327 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.276265 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d"} err="failed to get container status \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": rpc error: code = NotFound desc = could not find container \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": container with ID starting with 43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d not found: ID does not exist" Oct 30 18:38:17.276327 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.276285 5959 scope.go:122] "RemoveContainer" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" Oct 30 18:38:17.276718 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:17.276695 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": container with ID starting with da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d not found: ID does not exist" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" Oct 30 18:38:17.276797 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.276728 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d"} err="failed to get container status \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": rpc error: code = NotFound desc = could not find container \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": container with ID starting with da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d not found: ID does not exist" Oct 30 18:38:17.276797 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.276747 5959 scope.go:122] "RemoveContainer" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" Oct 30 18:38:17.277107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.277084 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626"} err="failed to get container status \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": rpc error: code = NotFound desc = could not find container \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": container with ID starting with 8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626 not found: ID does not exist" Oct 30 18:38:17.277212 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.277107 5959 scope.go:122] "RemoveContainer" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" Oct 30 18:38:17.277370 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.277352 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0"} err="failed to get container status \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": rpc error: code = NotFound desc = could not find container \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": container with ID starting with 9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0 not found: ID does not exist" Oct 30 18:38:17.277446 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.277370 5959 scope.go:122] "RemoveContainer" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" Oct 30 18:38:17.277728 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.277649 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c"} err="failed to get container status \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": rpc error: code = NotFound desc = could not find container \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": container with ID starting with f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c not found: ID does not exist" Oct 30 18:38:17.277728 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.277669 5959 scope.go:122] "RemoveContainer" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" Oct 30 18:38:17.278020 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.277924 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0"} err="failed to get container status \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": rpc error: code = NotFound desc = could not find container \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": container with ID starting with d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0 not found: ID does not exist" Oct 30 18:38:17.278020 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.277948 5959 scope.go:122] "RemoveContainer" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" Oct 30 18:38:17.278251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.278227 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2"} err="failed to get container status \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": rpc error: code = NotFound desc = could not find container \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": container with ID starting with 71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2 not found: ID does not exist" Oct 30 18:38:17.278303 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.278254 5959 scope.go:122] "RemoveContainer" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" Oct 30 18:38:17.278498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.278477 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d"} err="failed to get container status \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": rpc error: code = NotFound desc = could not find container \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": container with ID starting with 43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d not found: ID does not exist" Oct 30 18:38:17.278498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.278497 5959 scope.go:122] "RemoveContainer" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" Oct 30 18:38:17.278818 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.278800 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d"} err="failed to get container status \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": rpc error: code = NotFound desc = could not find container \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": container with ID starting with da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d not found: ID does not exist" Oct 30 18:38:17.278869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.278819 5959 scope.go:122] "RemoveContainer" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" Oct 30 18:38:17.279072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.279054 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626"} err="failed to get container status \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": rpc error: code = NotFound desc = could not find container \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": container with ID starting with 8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626 not found: ID does not exist" Oct 30 18:38:17.279150 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.279071 5959 scope.go:122] "RemoveContainer" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" Oct 30 18:38:17.279355 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.279330 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0"} err="failed to get container status \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": rpc error: code = NotFound desc = could not find container \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": container with ID starting with 9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0 not found: ID does not exist" Oct 30 18:38:17.279419 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.279357 5959 scope.go:122] "RemoveContainer" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" Oct 30 18:38:17.279648 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.279596 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c"} err="failed to get container status \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": rpc error: code = NotFound desc = could not find container \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": container with ID starting with f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c not found: ID does not exist" Oct 30 18:38:17.279648 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.279648 5959 scope.go:122] "RemoveContainer" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" Oct 30 18:38:17.280037 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.279994 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0"} err="failed to get container status \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": rpc error: code = NotFound desc = could not find container \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": container with ID starting with d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0 not found: ID does not exist" Oct 30 18:38:17.280154 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.280037 5959 scope.go:122] "RemoveContainer" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" Oct 30 18:38:17.280358 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.280271 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2"} err="failed to get container status \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": rpc error: code = NotFound desc = could not find container \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": container with ID starting with 71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2 not found: ID does not exist" Oct 30 18:38:17.280358 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.280292 5959 scope.go:122] "RemoveContainer" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" Oct 30 18:38:17.280651 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.280573 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d"} err="failed to get container status \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": rpc error: code = NotFound desc = could not find container \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": container with ID starting with 43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d not found: ID does not exist" Oct 30 18:38:17.280651 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.280630 5959 scope.go:122] "RemoveContainer" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" Oct 30 18:38:17.280914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.280893 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d"} err="failed to get container status \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": rpc error: code = NotFound desc = could not find container \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": container with ID starting with da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d not found: ID does not exist" Oct 30 18:38:17.280986 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.280915 5959 scope.go:122] "RemoveContainer" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" Oct 30 18:38:17.281199 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.281178 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626"} err="failed to get container status \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": rpc error: code = NotFound desc = could not find container \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": container with ID starting with 8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626 not found: ID does not exist" Oct 30 18:38:17.281264 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.281200 5959 scope.go:122] "RemoveContainer" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" Oct 30 18:38:17.281459 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.281438 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0"} err="failed to get container status \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": rpc error: code = NotFound desc = could not find container \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": container with ID starting with 9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0 not found: ID does not exist" Oct 30 18:38:17.281522 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.281460 5959 scope.go:122] "RemoveContainer" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" Oct 30 18:38:17.281760 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.281739 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c"} err="failed to get container status \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": rpc error: code = NotFound desc = could not find container \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": container with ID starting with f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c not found: ID does not exist" Oct 30 18:38:17.281760 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.281760 5959 scope.go:122] "RemoveContainer" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" Oct 30 18:38:17.282020 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.282001 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0"} err="failed to get container status \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": rpc error: code = NotFound desc = could not find container \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": container with ID starting with d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0 not found: ID does not exist" Oct 30 18:38:17.282095 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.282022 5959 scope.go:122] "RemoveContainer" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" Oct 30 18:38:17.282228 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.282197 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2"} err="failed to get container status \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": rpc error: code = NotFound desc = could not find container \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": container with ID starting with 71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2 not found: ID does not exist" Oct 30 18:38:17.282228 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.282217 5959 scope.go:122] "RemoveContainer" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" Oct 30 18:38:17.282520 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.282494 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d"} err="failed to get container status \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": rpc error: code = NotFound desc = could not find container \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": container with ID starting with 43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d not found: ID does not exist" Oct 30 18:38:17.282591 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.282521 5959 scope.go:122] "RemoveContainer" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" Oct 30 18:38:17.282810 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.282776 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d"} err="failed to get container status \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": rpc error: code = NotFound desc = could not find container \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": container with ID starting with da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d not found: ID does not exist" Oct 30 18:38:17.282810 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.282805 5959 scope.go:122] "RemoveContainer" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" Oct 30 18:38:17.283148 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.283126 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626"} err="failed to get container status \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": rpc error: code = NotFound desc = could not find container \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": container with ID starting with 8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626 not found: ID does not exist" Oct 30 18:38:17.283238 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.283149 5959 scope.go:122] "RemoveContainer" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" Oct 30 18:38:17.283439 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.283409 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0"} err="failed to get container status \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": rpc error: code = NotFound desc = could not find container \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": container with ID starting with 9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0 not found: ID does not exist" Oct 30 18:38:17.283439 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.283428 5959 scope.go:122] "RemoveContainer" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" Oct 30 18:38:17.283563 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.283509 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.283723 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.283690 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c"} err="failed to get container status \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": rpc error: code = NotFound desc = could not find container \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": container with ID starting with f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c not found: ID does not exist" Oct 30 18:38:17.283723 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.283715 5959 scope.go:122] "RemoveContainer" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" Oct 30 18:38:17.284827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.284801 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0"} err="failed to get container status \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": rpc error: code = NotFound desc = could not find container \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": container with ID starting with d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0 not found: ID does not exist" Oct 30 18:38:17.284827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.284830 5959 scope.go:122] "RemoveContainer" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" Oct 30 18:38:17.285214 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.285116 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2"} err="failed to get container status \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": rpc error: code = NotFound desc = could not find container \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": container with ID starting with 71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2 not found: ID does not exist" Oct 30 18:38:17.285214 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.285145 5959 scope.go:122] "RemoveContainer" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" Oct 30 18:38:17.285403 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.285385 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d"} err="failed to get container status \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": rpc error: code = NotFound desc = could not find container \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": container with ID starting with 43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d not found: ID does not exist" Oct 30 18:38:17.285462 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.285412 5959 scope.go:122] "RemoveContainer" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" Oct 30 18:38:17.285750 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.285668 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d"} err="failed to get container status \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": rpc error: code = NotFound desc = could not find container \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": container with ID starting with da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d not found: ID does not exist" Oct 30 18:38:17.285750 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.285692 5959 scope.go:122] "RemoveContainer" containerID="8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626" Oct 30 18:38:17.286052 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.286021 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626"} err="failed to get container status \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": rpc error: code = NotFound desc = could not find container \"8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626\": container with ID starting with 8c97e521bed67581fe87ec40a6da2be87b05c1002d13f4f711a9055c05444626 not found: ID does not exist" Oct 30 18:38:17.286052 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.286047 5959 scope.go:122] "RemoveContainer" containerID="9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0" Oct 30 18:38:17.286356 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.286335 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0"} err="failed to get container status \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": rpc error: code = NotFound desc = could not find container \"9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0\": container with ID starting with 9711735f79a443a83e12ac835467a60fb36d8250c8da18dfeb90704835b348d0 not found: ID does not exist" Oct 30 18:38:17.286439 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.286358 5959 scope.go:122] "RemoveContainer" containerID="f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c" Oct 30 18:38:17.286672 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.286651 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c"} err="failed to get container status \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": rpc error: code = NotFound desc = could not find container \"f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c\": container with ID starting with f56e896a1ba8638e8086a04f6072a9bf4f236c3d88ed42596dd3e96e389e944c not found: ID does not exist" Oct 30 18:38:17.286754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.286674 5959 scope.go:122] "RemoveContainer" containerID="d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0" Oct 30 18:38:17.286991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.286967 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0"} err="failed to get container status \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": rpc error: code = NotFound desc = could not find container \"d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0\": container with ID starting with d067ae375ea4c53995358c8f79dfa74add408f1cbdb6308ecc70f12bd28448a0 not found: ID does not exist" Oct 30 18:38:17.287051 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.286991 5959 scope.go:122] "RemoveContainer" containerID="71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2" Oct 30 18:38:17.287211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.287194 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2"} err="failed to get container status \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": rpc error: code = NotFound desc = could not find container \"71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2\": container with ID starting with 71afd4cf53714952c2313ddfc64e571a24f16fad2f0c6b5e73d6e3e7e24f75a2 not found: ID does not exist" Oct 30 18:38:17.287283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.287214 5959 scope.go:122] "RemoveContainer" containerID="43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d" Oct 30 18:38:17.287499 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.287473 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d"} err="failed to get container status \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": rpc error: code = NotFound desc = could not find container \"43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d\": container with ID starting with 43376835e8615d933df13105485741fff7b840f2287012ab197d5a80beed976d not found: ID does not exist" Oct 30 18:38:17.287559 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.287502 5959 scope.go:122] "RemoveContainer" containerID="da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d" Oct 30 18:38:17.287827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.287805 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d"} err="failed to get container status \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": rpc error: code = NotFound desc = could not find container \"da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d\": container with ID starting with da8e562a5cb2ab800ae55a5267452a220e84978eeac92b0f850207d468326e6d not found: ID does not exist" Oct 30 18:38:17.289117 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.289099 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:38:17.292366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.292347 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:38:17.292561 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.292488 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:38:17.292680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.292577 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kqrcb\"" Oct 30 18:38:17.293374 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.293333 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:38:17.293485 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.293394 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:38:17.293485 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.293480 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:38:17.297299 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.297279 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:38:17.299371 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.299353 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:38:17.299371 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.299356 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:38:17.385251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385212 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-config\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.385693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385266 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f096784b-2749-4f7e-adb3-448300f1013c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.385693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385368 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.385693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385415 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f096784b-2749-4f7e-adb3-448300f1013c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.385693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385492 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.385693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385538 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.385693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385573 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.385693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385678 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386016 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385720 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386016 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385745 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386016 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385794 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncx28\" (UniqueName: \"kubernetes.io/projected/f096784b-2749-4f7e-adb3-448300f1013c-kube-api-access-ncx28\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386016 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385930 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386016 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.385968 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386202 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.386022 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386202 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.386052 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386202 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.386085 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386202 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.386117 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386202 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.386172 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386202 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.386192 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.386422 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.386216 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.390116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.390095 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:38:17.390116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.390099 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:38:17.390446 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.390427 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:38:17.390567 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.390474 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:38:17.390654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.390577 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:38:17.390654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.390589 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:38:17.390654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.390645 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8lvuvgi8ronlo\"" Oct 30 18:38:17.486938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.486848 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f096784b-2749-4f7e-adb3-448300f1013c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.486938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.486892 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.486938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.486921 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f096784b-2749-4f7e-adb3-448300f1013c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.486946 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.486981 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487007 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487042 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487072 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487095 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487125 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncx28\" (UniqueName: \"kubernetes.io/projected/f096784b-2749-4f7e-adb3-448300f1013c-kube-api-access-ncx28\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487155 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487187 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487222 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487249 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487282 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487302 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487336 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487365 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487403 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487438 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-config\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.487868 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.487765 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.488462 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.488292 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.488462 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.488308 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.489022 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.488989 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.491742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.489543 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.491742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.490286 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.491742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.491446 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.491742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.491576 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.493491 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.493459 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f096784b-2749-4f7e-adb3-448300f1013c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.493636 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.493544 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.493636 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.493570 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.493750 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.493636 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.494069 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.494046 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-config\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.494143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.494122 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f096784b-2749-4f7e-adb3-448300f1013c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.494208 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.494136 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.494208 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.494146 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.494337 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.494322 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f096784b-2749-4f7e-adb3-448300f1013c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.494575 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.494551 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.495766 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.495724 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f096784b-2749-4f7e-adb3-448300f1013c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.497880 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.497775 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncx28\" (UniqueName: \"kubernetes.io/projected/f096784b-2749-4f7e-adb3-448300f1013c-kube-api-access-ncx28\") pod \"prometheus-k8s-0\" (UID: \"f096784b-2749-4f7e-adb3-448300f1013c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.614017 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.613975 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:17.796106 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.796024 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8565b888-5ed6-4d0f-a91c-87659bedfe00" path="/var/lib/kubelet/pods/8565b888-5ed6-4d0f-a91c-87659bedfe00/volumes" Oct 30 18:38:17.912680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:17.912561 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:38:17.919026 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:38:17.918999 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf096784b_2749_4f7e_adb3_448300f1013c.slice/crio-233d958d5620898dc09db4812fde1aa63cf589dec36d333cc3d97eefca8f8d35 WatchSource:0}: Error finding container 233d958d5620898dc09db4812fde1aa63cf589dec36d333cc3d97eefca8f8d35: Status 404 returned error can't find the container with id 233d958d5620898dc09db4812fde1aa63cf589dec36d333cc3d97eefca8f8d35 Oct 30 18:38:18.212365 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.212320 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" event={"ID":"d144b776-a2c9-450c-b189-e299d6d447f6","Type":"ContainerStarted","Data":"9ae30c3c336a3ea1ea7810e327abe03329129369526a833862bea2b3f3385f26"} Oct 30 18:38:18.212565 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.212395 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" event={"ID":"d144b776-a2c9-450c-b189-e299d6d447f6","Type":"ContainerStarted","Data":"114444ff9ddfc0299f5cbdbf8b90eccae507f9d28af52cc9153c6d5cb6bb63c5"} Oct 30 18:38:18.212565 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.212422 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" event={"ID":"d144b776-a2c9-450c-b189-e299d6d447f6","Type":"ContainerStarted","Data":"b812af2ea8b3de5d8041eaff8fcad97035ff419e0ce2995cbe677bfc26e9b052"} Oct 30 18:38:18.213776 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.213751 5959 generic.go:356] "Generic (PLEG): container finished" podID="f096784b-2749-4f7e-adb3-448300f1013c" containerID="a37144d560d601dba644dafb9a5682f2feb3880a0f4bc68fc47cfa652869100f" exitCode=0 Oct 30 18:38:18.213878 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.213836 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f096784b-2749-4f7e-adb3-448300f1013c","Type":"ContainerDied","Data":"a37144d560d601dba644dafb9a5682f2feb3880a0f4bc68fc47cfa652869100f"} Oct 30 18:38:18.213936 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.213922 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f096784b-2749-4f7e-adb3-448300f1013c","Type":"ContainerStarted","Data":"233d958d5620898dc09db4812fde1aa63cf589dec36d333cc3d97eefca8f8d35"} Oct 30 18:38:18.282291 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.282247 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5784486569-4gvpd" podStartSLOduration=1.684581418 podStartE2EDuration="3.282233935s" totalImagesPullingTime="1.597652517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:18.240756169 +0000 UTC m=+78.569167565" watchObservedRunningTime="2026-10-30 18:38:18.282233935 +0000 UTC m=+78.610645319" Oct 30 18:38:18.362786 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.362755 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f8b4478ff-kxwcj"] Oct 30 18:38:18.369093 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.369068 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.383654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.383627 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8b4478ff-kxwcj"] Oct 30 18:38:18.395339 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.395299 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-service-ca\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.395743 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.395366 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-oauth-serving-cert\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.395743 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.395396 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-serving-cert\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.395743 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.395535 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnkm\" (UniqueName: \"kubernetes.io/projected/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-kube-api-access-xjnkm\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.395743 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.395629 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-tmp\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.395743 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.395658 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-trusted-ca-bundle\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.395743 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.395688 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-oauth-config\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.396047 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.395781 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-config\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.496155 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.496128 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnkm\" (UniqueName: \"kubernetes.io/projected/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-kube-api-access-xjnkm\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.496294 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.496173 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-tmp\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.496294 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.496201 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-trusted-ca-bundle\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.496475 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.496452 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-oauth-config\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.496544 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.496506 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-config\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.496787 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.496571 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-service-ca\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.496787 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.496704 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-tmp\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.496858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.496784 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-oauth-serving-cert\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.497056 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.497037 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-trusted-ca-bundle\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.497415 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.497390 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-service-ca\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.497474 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.497405 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-oauth-serving-cert\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.497474 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.497430 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-serving-cert\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.498142 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.498116 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-config\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.500174 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.500148 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-oauth-config\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.500250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.500189 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-serving-cert\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.515708 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.515593 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnkm\" (UniqueName: \"kubernetes.io/projected/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-kube-api-access-xjnkm\") pod \"console-7f8b4478ff-kxwcj\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.697403 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.697365 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:18.982379 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:18.982342 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8b4478ff-kxwcj"] Oct 30 18:38:18.985550 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:38:18.985520 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd773ac2_0f2b_4827_8fc4_6ea5466c95b3.slice/crio-63d71418ddd508eb95c17055e160d52b51b204bcc834eac4775122e7cadf8205 WatchSource:0}: Error finding container 63d71418ddd508eb95c17055e160d52b51b204bcc834eac4775122e7cadf8205: Status 404 returned error can't find the container with id 63d71418ddd508eb95c17055e160d52b51b204bcc834eac4775122e7cadf8205 Oct 30 18:38:19.221161 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.221056 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8b4478ff-kxwcj" event={"ID":"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3","Type":"ContainerStarted","Data":"9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19"} Oct 30 18:38:19.221161 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.221120 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8b4478ff-kxwcj" event={"ID":"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3","Type":"ContainerStarted","Data":"63d71418ddd508eb95c17055e160d52b51b204bcc834eac4775122e7cadf8205"} Oct 30 18:38:19.225250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.225216 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f096784b-2749-4f7e-adb3-448300f1013c","Type":"ContainerStarted","Data":"10d097fd2427c618faa0dbf94f80916153cdfe22274282eb4595a659fe0814a1"} Oct 30 18:38:19.225390 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.225297 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f096784b-2749-4f7e-adb3-448300f1013c","Type":"ContainerStarted","Data":"3869971e69f46bea416f08550fd88772776e567d29dc1c3f7029fb5aaa5b4d70"} Oct 30 18:38:19.225390 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.225323 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f096784b-2749-4f7e-adb3-448300f1013c","Type":"ContainerStarted","Data":"8643d67e8039ee49438df72112d2b5bfbf62e54fc474d82036667cd713e2e572"} Oct 30 18:38:19.225390 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.225345 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f096784b-2749-4f7e-adb3-448300f1013c","Type":"ContainerStarted","Data":"7db1f1817c004df22b1a6f6d9e46a7df7ecbc60076cb0985f8265e409d2dac4d"} Oct 30 18:38:19.225390 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.225385 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f096784b-2749-4f7e-adb3-448300f1013c","Type":"ContainerStarted","Data":"6c03f8d19b35da79cc3503f6e5553414bb19e492c7a2acdbfeb9beb6ca7ca47d"} Oct 30 18:38:19.225533 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.225413 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f096784b-2749-4f7e-adb3-448300f1013c","Type":"ContainerStarted","Data":"f2b1b252d92605b40cf2bf9d26e1084e707a4ab3c51a98902edf0d953a648418"} Oct 30 18:38:19.249394 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.249313 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f8b4478ff-kxwcj" podStartSLOduration=1.249292206 podStartE2EDuration="1.249292206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:19.248101939 +0000 UTC m=+79.576513356" watchObservedRunningTime="2026-10-30 18:38:19.249292206 +0000 UTC m=+79.577703591" Oct 30 18:38:19.287983 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:19.287913 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.267445334 podStartE2EDuration="2.287897123s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.451789ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:19.287376275 +0000 UTC m=+79.615787661" watchObservedRunningTime="2026-10-30 18:38:19.287897123 +0000 UTC m=+79.616308508" Oct 30 18:38:22.614707 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:22.614667 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:38:27.779314 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:27.779279 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:38:28.697646 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:28.697584 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:28.697888 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:28.697679 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:28.703459 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:28.703434 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:29.270985 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:29.270950 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:38:29.321493 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:29.321463 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-668d98c4ff-w5c7v"] Oct 30 18:38:54.342082 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.341978 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-668d98c4ff-w5c7v" podUID="671f39e7-0d53-41ac-af9a-750a01c38f71" containerName="console" containerID="cri-o://7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde" gracePeriod=15 Oct 30 18:38:54.581698 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.581674 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:38:54.746537 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746503 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert\") pod \"671f39e7-0d53-41ac-af9a-750a01c38f71\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " Oct 30 18:38:54.746764 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746545 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle\") pod \"671f39e7-0d53-41ac-af9a-750a01c38f71\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " Oct 30 18:38:54.746764 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746562 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config\") pod \"671f39e7-0d53-41ac-af9a-750a01c38f71\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " Oct 30 18:38:54.746764 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746660 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca\") pod \"671f39e7-0d53-41ac-af9a-750a01c38f71\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " Oct 30 18:38:54.746764 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746728 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert\") pod \"671f39e7-0d53-41ac-af9a-750a01c38f71\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " Oct 30 18:38:54.746764 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746762 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm\" (UniqueName: \"kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm\") pod \"671f39e7-0d53-41ac-af9a-750a01c38f71\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " Oct 30 18:38:54.747039 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746792 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config\") pod \"671f39e7-0d53-41ac-af9a-750a01c38f71\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " Oct 30 18:38:54.747039 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746822 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp\" (UniqueName: \"kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp\") pod \"671f39e7-0d53-41ac-af9a-750a01c38f71\" (UID: \"671f39e7-0d53-41ac-af9a-750a01c38f71\") " Oct 30 18:38:54.747039 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.746991 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle" pod "671f39e7-0d53-41ac-af9a-750a01c38f71" (UID: "671f39e7-0d53-41ac-af9a-750a01c38f71"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.747193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.747038 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca" pod "671f39e7-0d53-41ac-af9a-750a01c38f71" (UID: "671f39e7-0d53-41ac-af9a-750a01c38f71"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.747193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.747123 5959 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-trusted-ca-bundle\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:54.747297 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.747258 5959 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:54.747430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.747388 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert" pod "671f39e7-0d53-41ac-af9a-750a01c38f71" (UID: "671f39e7-0d53-41ac-af9a-750a01c38f71"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.747627 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.747441 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config" pod "671f39e7-0d53-41ac-af9a-750a01c38f71" (UID: "671f39e7-0d53-41ac-af9a-750a01c38f71"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.747627 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.747569 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp" pod "671f39e7-0d53-41ac-af9a-750a01c38f71" (UID: "671f39e7-0d53-41ac-af9a-750a01c38f71"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.749086 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.749061 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config" pod "671f39e7-0d53-41ac-af9a-750a01c38f71" (UID: "671f39e7-0d53-41ac-af9a-750a01c38f71"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:54.749192 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.749141 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert" pod "671f39e7-0d53-41ac-af9a-750a01c38f71" (UID: "671f39e7-0d53-41ac-af9a-750a01c38f71"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:54.749247 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.749197 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm" pod "671f39e7-0d53-41ac-af9a-750a01c38f71" (UID: "671f39e7-0d53-41ac-af9a-750a01c38f71"). InnerVolumeSpecName "kube-api-access-wqckm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.848199 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.848162 5959 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-oauth-serving-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:54.848199 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.848193 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqckm\" (UniqueName: \"kubernetes.io/projected/671f39e7-0d53-41ac-af9a-750a01c38f71-kube-api-access-wqckm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:54.848199 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.848206 5959 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/671f39e7-0d53-41ac-af9a-750a01c38f71-console-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:54.848426 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.848216 5959 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671f39e7-0d53-41ac-af9a-750a01c38f71-tmp\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:54.848426 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.848226 5959 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-serving-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:54.848426 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:54.848234 5959 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/671f39e7-0d53-41ac-af9a-750a01c38f71-console-oauth-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:38:55.368742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.368703 5959 generic.go:356] "Generic (PLEG): container finished" podID="671f39e7-0d53-41ac-af9a-750a01c38f71" containerID="7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde" exitCode=0 Oct 30 18:38:55.369193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.368757 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668d98c4ff-w5c7v" Oct 30 18:38:55.369193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.368786 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668d98c4ff-w5c7v" event={"ID":"671f39e7-0d53-41ac-af9a-750a01c38f71","Type":"ContainerDied","Data":"7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde"} Oct 30 18:38:55.369193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.368842 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668d98c4ff-w5c7v" event={"ID":"671f39e7-0d53-41ac-af9a-750a01c38f71","Type":"ContainerDied","Data":"2bf20d347dfa51b6449cf1355defaec52e74b130b6bd1a23a5c77e553d5d398f"} Oct 30 18:38:55.369193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.368865 5959 scope.go:122] "RemoveContainer" containerID="7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde" Oct 30 18:38:55.377631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.377591 5959 scope.go:122] "RemoveContainer" containerID="7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde" Oct 30 18:38:55.377912 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:55.377892 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde\": container with ID starting with 7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde not found: ID does not exist" containerID="7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde" Oct 30 18:38:55.377979 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.377922 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde"} err="failed to get container status \"7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde\": rpc error: code = NotFound desc = could not find container \"7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde\": container with ID starting with 7a3adc258697b4d9fd0ba3ddf0f3b06395259b21795873dae412223b20a5bcde not found: ID does not exist" Oct 30 18:38:55.389429 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.389401 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-668d98c4ff-w5c7v"] Oct 30 18:38:55.392133 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.392112 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-668d98c4ff-w5c7v"] Oct 30 18:38:55.794176 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:38:55.794139 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="671f39e7-0d53-41ac-af9a-750a01c38f71" path="/var/lib/kubelet/pods/671f39e7-0d53-41ac-af9a-750a01c38f71/volumes" Oct 30 18:38:57.779155 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:38:57.779117 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:39:17.614827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:17.614785 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:39:17.631544 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:17.631513 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:39:18.471600 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:18.471570 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:39:27.779788 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:39:27.779746 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:39:30.995885 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:30.995845 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67577fc95c-4n59h"] Oct 30 18:39:30.996377 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:30.996352 5959 cpu_manager.go:414] "containerMap: removing container" podUID="671f39e7-0d53-41ac-af9a-750a01c38f71" containerName="console" Oct 30 18:39:30.996377 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:30.996371 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="671f39e7-0d53-41ac-af9a-750a01c38f71" containerName="console" Oct 30 18:39:30.996498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:30.996447 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="671f39e7-0d53-41ac-af9a-750a01c38f71" containerName="console" Oct 30 18:39:31.000333 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.000313 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.016988 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.016959 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgr7\" (UniqueName: \"kubernetes.io/projected/65da6205-b5cb-423f-b270-5122dd5b502b-kube-api-access-nvgr7\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.017097 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.016994 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65da6205-b5cb-423f-b270-5122dd5b502b-console-serving-cert\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.017097 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.017028 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-console-config\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.017172 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.017093 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-oauth-serving-cert\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.017172 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.017121 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-service-ca\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.017233 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.017191 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-trusted-ca-bundle\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.017233 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.017224 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65da6205-b5cb-423f-b270-5122dd5b502b-console-oauth-config\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.017296 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.017256 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65da6205-b5cb-423f-b270-5122dd5b502b-tmp\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.023881 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.023852 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67577fc95c-4n59h"] Oct 30 18:39:31.118093 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118053 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-oauth-serving-cert\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118099 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-service-ca\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118129 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-trusted-ca-bundle\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118151 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65da6205-b5cb-423f-b270-5122dd5b502b-console-oauth-config\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118189 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65da6205-b5cb-423f-b270-5122dd5b502b-tmp\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118245 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgr7\" (UniqueName: \"kubernetes.io/projected/65da6205-b5cb-423f-b270-5122dd5b502b-kube-api-access-nvgr7\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118522 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118276 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65da6205-b5cb-423f-b270-5122dd5b502b-console-serving-cert\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118522 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118310 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-console-config\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118743 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118709 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65da6205-b5cb-423f-b270-5122dd5b502b-tmp\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.118948 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.118915 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-service-ca\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.119199 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.119063 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-console-config\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.119287 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.119261 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-trusted-ca-bundle\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.119458 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.119440 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65da6205-b5cb-423f-b270-5122dd5b502b-oauth-serving-cert\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.120898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.120872 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65da6205-b5cb-423f-b270-5122dd5b502b-console-oauth-config\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.121010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.120986 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65da6205-b5cb-423f-b270-5122dd5b502b-console-serving-cert\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.126572 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.126550 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgr7\" (UniqueName: \"kubernetes.io/projected/65da6205-b5cb-423f-b270-5122dd5b502b-kube-api-access-nvgr7\") pod \"console-67577fc95c-4n59h\" (UID: \"65da6205-b5cb-423f-b270-5122dd5b502b\") " pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.311700 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.311617 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:31.585562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:31.585484 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67577fc95c-4n59h"] Oct 30 18:39:31.588576 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:39:31.588554 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65da6205_b5cb_423f_b270_5122dd5b502b.slice/crio-f3798e6e0c2b58b19c7ebe054f1c4aaabb7ae3583196c6ff8b3e95dbc588dd69 WatchSource:0}: Error finding container f3798e6e0c2b58b19c7ebe054f1c4aaabb7ae3583196c6ff8b3e95dbc588dd69: Status 404 returned error can't find the container with id f3798e6e0c2b58b19c7ebe054f1c4aaabb7ae3583196c6ff8b3e95dbc588dd69 Oct 30 18:39:32.508638 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:32.508589 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67577fc95c-4n59h" event={"ID":"65da6205-b5cb-423f-b270-5122dd5b502b","Type":"ContainerStarted","Data":"092f49268cecb3775e8e2570626efd6c4007de689f6c09a9347c12bc0e5f0d49"} Oct 30 18:39:32.508638 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:32.508652 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67577fc95c-4n59h" event={"ID":"65da6205-b5cb-423f-b270-5122dd5b502b","Type":"ContainerStarted","Data":"f3798e6e0c2b58b19c7ebe054f1c4aaabb7ae3583196c6ff8b3e95dbc588dd69"} Oct 30 18:39:32.531848 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:32.531798 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67577fc95c-4n59h" podStartSLOduration=2.531784019 podStartE2EDuration="2.531784019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:32.529392661 +0000 UTC m=+152.857804042" watchObservedRunningTime="2026-10-30 18:39:32.531784019 +0000 UTC m=+152.860195463" Oct 30 18:39:41.312728 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:41.312686 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:41.313229 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:41.312749 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:41.317476 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:41.317452 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:41.552444 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:41.552419 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67577fc95c-4n59h" Oct 30 18:39:41.598801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:39:41.598717 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f8b4478ff-kxwcj"] Oct 30 18:39:57.779232 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:39:57.779193 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:40:06.619402 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:06.619343 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f8b4478ff-kxwcj" podUID="dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" containerName="console" containerID="cri-o://9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19" gracePeriod=15 Oct 30 18:40:06.868153 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:06.868120 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:40:07.036960 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.036930 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-trusted-ca-bundle\") pod \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " Oct 30 18:40:07.037147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.036975 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-kube-api-access-xjnkm\" (UniqueName: \"kubernetes.io/projected/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-kube-api-access-xjnkm\") pod \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " Oct 30 18:40:07.037147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037011 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-tmp\") pod \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " Oct 30 18:40:07.037147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037073 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-serving-cert\") pod \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " Oct 30 18:40:07.037147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037090 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-oauth-config\") pod \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " Oct 30 18:40:07.037147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037111 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-config\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-config\") pod \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " Oct 30 18:40:07.037401 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037148 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-oauth-serving-cert\") pod \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " Oct 30 18:40:07.037401 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037200 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-service-ca\") pod \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\" (UID: \"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3\") " Oct 30 18:40:07.037534 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037497 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-trusted-ca-bundle" pod "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" (UID: "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:07.037596 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037569 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-tmp" pod "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" (UID: "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:07.037686 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037656 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-oauth-serving-cert" pod "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" (UID: "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:07.037768 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037741 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-config" pod "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" (UID: "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:07.037805 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.037754 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-service-ca" pod "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" (UID: "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:07.039291 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.039262 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-serving-cert" pod "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" (UID: "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:07.039291 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.039265 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-kube-api-access-xjnkm" pod "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" (UID: "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3"). InnerVolumeSpecName "kube-api-access-xjnkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:07.039491 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.039470 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-oauth-config" pod "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" (UID: "dd773ac2-0f2b-4827-8fc4-6ea5466c95b3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:07.138024 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.137989 5959 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:07.138024 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.138018 5959 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-trusted-ca-bundle\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:07.138024 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.138028 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnkm\" (UniqueName: \"kubernetes.io/projected/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-kube-api-access-xjnkm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:07.138241 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.138039 5959 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-tmp\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:07.138241 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.138048 5959 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-serving-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:07.138241 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.138056 5959 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-oauth-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:07.138241 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.138064 5959 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-console-config\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:07.138241 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.138072 5959 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3-oauth-serving-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:07.649955 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.649920 5959 generic.go:356] "Generic (PLEG): container finished" podID="dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" containerID="9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19" exitCode=0 Oct 30 18:40:07.650399 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.649972 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8b4478ff-kxwcj" Oct 30 18:40:07.650399 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.649972 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8b4478ff-kxwcj" event={"ID":"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3","Type":"ContainerDied","Data":"9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19"} Oct 30 18:40:07.650399 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.650025 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8b4478ff-kxwcj" event={"ID":"dd773ac2-0f2b-4827-8fc4-6ea5466c95b3","Type":"ContainerDied","Data":"63d71418ddd508eb95c17055e160d52b51b204bcc834eac4775122e7cadf8205"} Oct 30 18:40:07.650399 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.650052 5959 scope.go:122] "RemoveContainer" containerID="9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19" Oct 30 18:40:07.658622 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.658585 5959 scope.go:122] "RemoveContainer" containerID="9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19" Oct 30 18:40:07.658959 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:40:07.658939 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19\": container with ID starting with 9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19 not found: ID does not exist" containerID="9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19" Oct 30 18:40:07.659037 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.658966 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19"} err="failed to get container status \"9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19\": rpc error: code = NotFound desc = could not find container \"9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19\": container with ID starting with 9b9b044f1ac397ccdb01228756456575000a3ef4f26415c57bef884bf0530b19 not found: ID does not exist" Oct 30 18:40:07.669762 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.669737 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f8b4478ff-kxwcj"] Oct 30 18:40:07.673531 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.673512 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f8b4478ff-kxwcj"] Oct 30 18:40:07.794073 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:40:07.794043 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" path="/var/lib/kubelet/pods/dd773ac2-0f2b-4827-8fc4-6ea5466c95b3/volumes" Oct 30 18:40:27.779127 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:40:27.779088 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:40:57.778786 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:40:57.778744 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:41:27.778312 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:41:27.778270 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:41:57.779083 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:41:57.778994 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:42:27.778516 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:42:27.778477 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:42:50.005323 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.005292 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj"] Oct 30 18:42:50.005850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.005661 5959 cpu_manager.go:414] "containerMap: removing container" podUID="dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" containerName="console" Oct 30 18:42:50.005850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.005673 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" containerName="console" Oct 30 18:42:50.005850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.005742 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd773ac2-0f2b-4827-8fc4-6ea5466c95b3" containerName="console" Oct 30 18:42:50.009657 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.009629 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.046111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.046077 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj"] Oct 30 18:42:50.046278 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.046258 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7p6p\" (UniqueName: \"kubernetes.io/projected/edb91a74-e403-4e22-a058-740da77a9069-kube-api-access-t7p6p\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.046419 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.046296 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.046419 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.046355 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edb91a74-e403-4e22-a058-740da77a9069-manifests\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.046539 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.046415 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.046596 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.046574 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb91a74-e403-4e22-a058-740da77a9069-tmp\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.046683 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.046599 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.082041 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.082013 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:42:50.082236 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.082218 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:42:50.082384 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.082370 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:42:50.082463 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.082378 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2vxg7\"" Oct 30 18:42:50.082463 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.082455 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:42:50.082551 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.082463 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:42:50.147519 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.147489 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edb91a74-e403-4e22-a058-740da77a9069-manifests\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.147519 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.147518 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.147752 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.147548 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb91a74-e403-4e22-a058-740da77a9069-tmp\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.147752 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.147660 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.147908 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.147883 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7p6p\" (UniqueName: \"kubernetes.io/projected/edb91a74-e403-4e22-a058-740da77a9069-kube-api-access-t7p6p\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.147969 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.147930 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.148025 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.147985 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edb91a74-e403-4e22-a058-740da77a9069-manifests\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.150179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.150155 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.150396 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.150372 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.150512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.150429 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb91a74-e403-4e22-a058-740da77a9069-tmp\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.150669 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.150651 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edb91a74-e403-4e22-a058-740da77a9069-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.157853 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.157833 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7p6p\" (UniqueName: \"kubernetes.io/projected/edb91a74-e403-4e22-a058-740da77a9069-kube-api-access-t7p6p\") pod \"opendatahub-operator-controller-manager-8ff44d5d-nt2zj\" (UID: \"edb91a74-e403-4e22-a058-740da77a9069\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.329566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.329483 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:50.647998 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:50.647971 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj"] Oct 30 18:42:50.649585 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:42:50.649544 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb91a74_e403_4e22_a058_740da77a9069.slice/crio-3b48a5fb7855f577ba1c83aeb08eca059f7cf1af66ab02abe933953f5e717b7b WatchSource:0}: Error finding container 3b48a5fb7855f577ba1c83aeb08eca059f7cf1af66ab02abe933953f5e717b7b: Status 404 returned error can't find the container with id 3b48a5fb7855f577ba1c83aeb08eca059f7cf1af66ab02abe933953f5e717b7b Oct 30 18:42:51.266181 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:51.266123 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" event={"ID":"edb91a74-e403-4e22-a058-740da77a9069","Type":"ContainerStarted","Data":"3b48a5fb7855f577ba1c83aeb08eca059f7cf1af66ab02abe933953f5e717b7b"} Oct 30 18:42:54.281680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:54.281641 5959 generic.go:356] "Generic (PLEG): container finished" podID="edb91a74-e403-4e22-a058-740da77a9069" containerID="47a8bf89fb86859a226d45680218a04e64282d8c832470d75a3b316c8dd9a61c" exitCode=0 Oct 30 18:42:54.282143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:54.281707 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" event={"ID":"edb91a74-e403-4e22-a058-740da77a9069","Type":"ContainerDied","Data":"47a8bf89fb86859a226d45680218a04e64282d8c832470d75a3b316c8dd9a61c"} Oct 30 18:42:55.287249 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.287215 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" event={"ID":"edb91a74-e403-4e22-a058-740da77a9069","Type":"ContainerStarted","Data":"660ae334fba72d7260fa113c8b34022cdaf9cc742a7907321e0ddd32075ae632"} Oct 30 18:42:55.287688 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.287337 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:42:55.570348 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.570254 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" podStartSLOduration=2.779903341 podStartE2EDuration="6.570237571s" totalImagesPullingTime="3.370727546s" totalInitContainerRuntime="419.606684ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:55.342567054 +0000 UTC m=+355.670978441" watchObservedRunningTime="2026-10-30 18:42:55.570237571 +0000 UTC m=+355.898648955" Oct 30 18:42:55.570808 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.570791 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6"] Oct 30 18:42:55.574315 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.574300 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" Oct 30 18:42:55.580355 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.580330 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 18:42:55.580487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.580398 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:42:55.580822 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.580695 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-pvt22\"" Oct 30 18:42:55.587517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.587491 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6"] Oct 30 18:42:55.593763 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.593742 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mwk\" (UniqueName: \"kubernetes.io/projected/9d86e502-ce04-4743-b0e3-c07d840ca395-kube-api-access-z2mwk\") pod \"jobset-operator-674cd59dc5-fr8r6\" (UID: \"9d86e502-ce04-4743-b0e3-c07d840ca395\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" Oct 30 18:42:55.593860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.593770 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d86e502-ce04-4743-b0e3-c07d840ca395-tmp\") pod \"jobset-operator-674cd59dc5-fr8r6\" (UID: \"9d86e502-ce04-4743-b0e3-c07d840ca395\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" Oct 30 18:42:55.694953 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.694917 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mwk\" (UniqueName: \"kubernetes.io/projected/9d86e502-ce04-4743-b0e3-c07d840ca395-kube-api-access-z2mwk\") pod \"jobset-operator-674cd59dc5-fr8r6\" (UID: \"9d86e502-ce04-4743-b0e3-c07d840ca395\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" Oct 30 18:42:55.694953 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.694956 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d86e502-ce04-4743-b0e3-c07d840ca395-tmp\") pod \"jobset-operator-674cd59dc5-fr8r6\" (UID: \"9d86e502-ce04-4743-b0e3-c07d840ca395\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" Oct 30 18:42:55.695351 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.695332 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d86e502-ce04-4743-b0e3-c07d840ca395-tmp\") pod \"jobset-operator-674cd59dc5-fr8r6\" (UID: \"9d86e502-ce04-4743-b0e3-c07d840ca395\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" Oct 30 18:42:55.703922 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.703899 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mwk\" (UniqueName: \"kubernetes.io/projected/9d86e502-ce04-4743-b0e3-c07d840ca395-kube-api-access-z2mwk\") pod \"jobset-operator-674cd59dc5-fr8r6\" (UID: \"9d86e502-ce04-4743-b0e3-c07d840ca395\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" Oct 30 18:42:55.894013 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:55.893984 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" Oct 30 18:42:56.177037 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:56.176957 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6"] Oct 30 18:42:56.180820 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:42:56.180795 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d86e502_ce04_4743_b0e3_c07d840ca395.slice/crio-07eb4a48f11aefe9eb438a06aa084605416f5870c8150cb4f152e26282670f72 WatchSource:0}: Error finding container 07eb4a48f11aefe9eb438a06aa084605416f5870c8150cb4f152e26282670f72: Status 404 returned error can't find the container with id 07eb4a48f11aefe9eb438a06aa084605416f5870c8150cb4f152e26282670f72 Oct 30 18:42:56.292641 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:56.292583 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" event={"ID":"9d86e502-ce04-4743-b0e3-c07d840ca395","Type":"ContainerStarted","Data":"07eb4a48f11aefe9eb438a06aa084605416f5870c8150cb4f152e26282670f72"} Oct 30 18:42:57.790495 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:42:57.790462 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:42:59.307336 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:59.307300 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" event={"ID":"9d86e502-ce04-4743-b0e3-c07d840ca395","Type":"ContainerStarted","Data":"bdd562f9ae36261aed8c2f0bb9e3895bb9c74dd67c7f2525e02fb535041f6b40"} Oct 30 18:42:59.327472 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:42:59.327409 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fr8r6" podStartSLOduration=1.340664179 podStartE2EDuration="4.327390194s" totalImagesPullingTime="2.986726015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:59.326766046 +0000 UTC m=+359.655177436" watchObservedRunningTime="2026-10-30 18:42:59.327390194 +0000 UTC m=+359.655801580" Oct 30 18:43:06.295397 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:06.295361 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-nt2zj" Oct 30 18:43:10.584635 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.584540 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2"] Oct 30 18:43:10.592793 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.592768 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:10.617049 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.617011 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:43:10.617049 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.617011 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:43:10.619023 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.619004 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4r76p\"" Oct 30 18:43:10.621113 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.621082 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2"] Oct 30 18:43:10.740702 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.740666 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rff49\" (UniqueName: \"kubernetes.io/projected/193f879e-f190-4a8d-8b3b-dcf68accc4ef-kube-api-access-rff49\") pod \"cert-manager-webhook-5dbbbf75cc-cjqh2\" (UID: \"193f879e-f190-4a8d-8b3b-dcf68accc4ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:10.740926 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.740738 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/193f879e-f190-4a8d-8b3b-dcf68accc4ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cjqh2\" (UID: \"193f879e-f190-4a8d-8b3b-dcf68accc4ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:10.842424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.842340 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rff49\" (UniqueName: \"kubernetes.io/projected/193f879e-f190-4a8d-8b3b-dcf68accc4ef-kube-api-access-rff49\") pod \"cert-manager-webhook-5dbbbf75cc-cjqh2\" (UID: \"193f879e-f190-4a8d-8b3b-dcf68accc4ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:10.842424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.842407 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/193f879e-f190-4a8d-8b3b-dcf68accc4ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cjqh2\" (UID: \"193f879e-f190-4a8d-8b3b-dcf68accc4ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:10.861302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.861262 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rff49\" (UniqueName: \"kubernetes.io/projected/193f879e-f190-4a8d-8b3b-dcf68accc4ef-kube-api-access-rff49\") pod \"cert-manager-webhook-5dbbbf75cc-cjqh2\" (UID: \"193f879e-f190-4a8d-8b3b-dcf68accc4ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:10.861579 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.861556 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/193f879e-f190-4a8d-8b3b-dcf68accc4ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cjqh2\" (UID: \"193f879e-f190-4a8d-8b3b-dcf68accc4ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:10.904664 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:10.904625 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:11.189962 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:11.189916 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2"] Oct 30 18:43:11.193404 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:43:11.193375 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod193f879e_f190_4a8d_8b3b_dcf68accc4ef.slice/crio-72526b0257e8feb28d3b730595fece316de729408fa5cb799de79e79f24d46e4 WatchSource:0}: Error finding container 72526b0257e8feb28d3b730595fece316de729408fa5cb799de79e79f24d46e4: Status 404 returned error can't find the container with id 72526b0257e8feb28d3b730595fece316de729408fa5cb799de79e79f24d46e4 Oct 30 18:43:11.364421 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:11.364380 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" event={"ID":"193f879e-f190-4a8d-8b3b-dcf68accc4ef","Type":"ContainerStarted","Data":"72526b0257e8feb28d3b730595fece316de729408fa5cb799de79e79f24d46e4"} Oct 30 18:43:15.388393 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:15.388358 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" event={"ID":"193f879e-f190-4a8d-8b3b-dcf68accc4ef","Type":"ContainerStarted","Data":"c04cf9fb807d2ec3d8c92812b84c4855d96e2aa26efb64250654b87d9d678efb"} Oct 30 18:43:15.388835 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:15.388466 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:15.418536 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:15.418480 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" podStartSLOduration=1.498091267 podStartE2EDuration="5.418464182s" totalImagesPullingTime="3.920372915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:15.417052975 +0000 UTC m=+375.745464356" watchObservedRunningTime="2026-10-30 18:43:15.418464182 +0000 UTC m=+375.746875567" Oct 30 18:43:17.351575 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.351531 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q"] Oct 30 18:43:17.355732 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.355700 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" Oct 30 18:43:17.369391 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.365150 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sbvxr\"" Oct 30 18:43:17.369391 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.366023 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q"] Oct 30 18:43:17.509442 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.509392 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/971510fe-ff21-4ca4-9524-d19757c775a8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mbp6q\" (UID: \"971510fe-ff21-4ca4-9524-d19757c775a8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" Oct 30 18:43:17.509442 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.509438 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhn5\" (UniqueName: \"kubernetes.io/projected/971510fe-ff21-4ca4-9524-d19757c775a8-kube-api-access-9zhn5\") pod \"cert-manager-cainjector-5c57bbd4fc-mbp6q\" (UID: \"971510fe-ff21-4ca4-9524-d19757c775a8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" Oct 30 18:43:17.604869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.604793 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9"] Oct 30 18:43:17.608307 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.608282 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.610934 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.610900 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/971510fe-ff21-4ca4-9524-d19757c775a8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mbp6q\" (UID: \"971510fe-ff21-4ca4-9524-d19757c775a8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" Oct 30 18:43:17.611084 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.610936 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhn5\" (UniqueName: \"kubernetes.io/projected/971510fe-ff21-4ca4-9524-d19757c775a8-kube-api-access-9zhn5\") pod \"cert-manager-cainjector-5c57bbd4fc-mbp6q\" (UID: \"971510fe-ff21-4ca4-9524-d19757c775a8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" Oct 30 18:43:17.619957 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.619915 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 18:43:17.620121 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.619991 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:43:17.620121 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.620105 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 18:43:17.620276 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.620259 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:43:17.620397 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.620379 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fzh8z\"" Oct 30 18:43:17.624377 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.624346 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9"] Oct 30 18:43:17.628663 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.628632 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhn5\" (UniqueName: \"kubernetes.io/projected/971510fe-ff21-4ca4-9524-d19757c775a8-kube-api-access-9zhn5\") pod \"cert-manager-cainjector-5c57bbd4fc-mbp6q\" (UID: \"971510fe-ff21-4ca4-9524-d19757c775a8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" Oct 30 18:43:17.630647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.630618 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/971510fe-ff21-4ca4-9524-d19757c775a8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mbp6q\" (UID: \"971510fe-ff21-4ca4-9524-d19757c775a8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" Oct 30 18:43:17.670945 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.670906 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" Oct 30 18:43:17.712453 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.712414 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5f538d4-ba0c-4f64-953d-5f779027b94c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.712453 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.712464 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955dr\" (UniqueName: \"kubernetes.io/projected/a5f538d4-ba0c-4f64-953d-5f779027b94c-kube-api-access-955dr\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.712774 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.712634 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5f538d4-ba0c-4f64-953d-5f779027b94c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.813369 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.813327 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5f538d4-ba0c-4f64-953d-5f779027b94c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.813584 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.813380 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-955dr\" (UniqueName: \"kubernetes.io/projected/a5f538d4-ba0c-4f64-953d-5f779027b94c-kube-api-access-955dr\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.813584 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.813439 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5f538d4-ba0c-4f64-953d-5f779027b94c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.814264 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.814242 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5f538d4-ba0c-4f64-953d-5f779027b94c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.816080 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.816056 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5f538d4-ba0c-4f64-953d-5f779027b94c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.827068 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.827037 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-955dr\" (UniqueName: \"kubernetes.io/projected/a5f538d4-ba0c-4f64-953d-5f779027b94c-kube-api-access-955dr\") pod \"kuadrant-console-plugin-66c4fb575d-crll9\" (UID: \"a5f538d4-ba0c-4f64-953d-5f779027b94c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.923538 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.923502 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" Oct 30 18:43:17.958528 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:17.956809 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q"] Oct 30 18:43:17.961168 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:43:17.960893 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod971510fe_ff21_4ca4_9524_d19757c775a8.slice/crio-9d4d3a65fb5d4f7c166c496539a8fa2ff88ec1f3aec832abd667d3d17fa8e66f WatchSource:0}: Error finding container 9d4d3a65fb5d4f7c166c496539a8fa2ff88ec1f3aec832abd667d3d17fa8e66f: Status 404 returned error can't find the container with id 9d4d3a65fb5d4f7c166c496539a8fa2ff88ec1f3aec832abd667d3d17fa8e66f Oct 30 18:43:18.107175 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.107135 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s"] Oct 30 18:43:18.110728 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.110703 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.124386 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.124356 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s"] Oct 30 18:43:18.187954 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.187876 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:43:18.188213 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.188192 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:43:18.204430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.204385 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 18:43:18.204578 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.204442 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:43:18.204678 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.204663 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nzddx\"" Oct 30 18:43:18.217305 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.217257 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f75d54-cda6-4732-aedd-1250c4aa2c57-tmp\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.217447 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.217335 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f75d54-cda6-4732-aedd-1250c4aa2c57-tls-certs\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.217525 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.217505 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcck9\" (UniqueName: \"kubernetes.io/projected/a6f75d54-cda6-4732-aedd-1250c4aa2c57-kube-api-access-mcck9\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.229161 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.229115 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9"] Oct 30 18:43:18.231953 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:43:18.231924 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f538d4_ba0c_4f64_953d_5f779027b94c.slice/crio-a4b69e3c3e1cec990216c175f4e7809297349eaf1c1c3f2df477896713a0490d WatchSource:0}: Error finding container a4b69e3c3e1cec990216c175f4e7809297349eaf1c1c3f2df477896713a0490d: Status 404 returned error can't find the container with id a4b69e3c3e1cec990216c175f4e7809297349eaf1c1c3f2df477896713a0490d Oct 30 18:43:18.233489 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.233469 5959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:43:18.318553 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.318506 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcck9\" (UniqueName: \"kubernetes.io/projected/a6f75d54-cda6-4732-aedd-1250c4aa2c57-kube-api-access-mcck9\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.318770 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.318572 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f75d54-cda6-4732-aedd-1250c4aa2c57-tmp\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.318770 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.318627 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f75d54-cda6-4732-aedd-1250c4aa2c57-tls-certs\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.321143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.321113 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f75d54-cda6-4732-aedd-1250c4aa2c57-tmp\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.321965 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.321940 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f75d54-cda6-4732-aedd-1250c4aa2c57-tls-certs\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.337225 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.337193 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcck9\" (UniqueName: \"kubernetes.io/projected/a6f75d54-cda6-4732-aedd-1250c4aa2c57-kube-api-access-mcck9\") pod \"kube-auth-proxy-6f89dc687f-h9g5s\" (UID: \"a6f75d54-cda6-4732-aedd-1250c4aa2c57\") " pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.406682 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.406641 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" event={"ID":"a5f538d4-ba0c-4f64-953d-5f779027b94c","Type":"ContainerStarted","Data":"a4b69e3c3e1cec990216c175f4e7809297349eaf1c1c3f2df477896713a0490d"} Oct 30 18:43:18.409508 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.409466 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" event={"ID":"971510fe-ff21-4ca4-9524-d19757c775a8","Type":"ContainerStarted","Data":"5ffc62a7b62ceee98c87822dfc81102f4fdebb6b2c8af4fa02b5406c5e099182"} Oct 30 18:43:18.409508 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.409514 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" event={"ID":"971510fe-ff21-4ca4-9524-d19757c775a8","Type":"ContainerStarted","Data":"9d4d3a65fb5d4f7c166c496539a8fa2ff88ec1f3aec832abd667d3d17fa8e66f"} Oct 30 18:43:18.429503 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.429464 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" Oct 30 18:43:18.440618 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.440498 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mbp6q" podStartSLOduration=1.4404777690000001 podStartE2EDuration="1.440477769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:18.43916916 +0000 UTC m=+378.767580546" watchObservedRunningTime="2026-10-30 18:43:18.440477769 +0000 UTC m=+378.768889155" Oct 30 18:43:18.731929 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:18.731835 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s"] Oct 30 18:43:18.736822 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:43:18.736785 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f75d54_cda6_4732_aedd_1250c4aa2c57.slice/crio-632430b3aec9a0b8f2002ebb67a7918cf5841c6ca13c80278659d7678b9f785c WatchSource:0}: Error finding container 632430b3aec9a0b8f2002ebb67a7918cf5841c6ca13c80278659d7678b9f785c: Status 404 returned error can't find the container with id 632430b3aec9a0b8f2002ebb67a7918cf5841c6ca13c80278659d7678b9f785c Oct 30 18:43:19.425540 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:19.424771 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" event={"ID":"a6f75d54-cda6-4732-aedd-1250c4aa2c57","Type":"ContainerStarted","Data":"632430b3aec9a0b8f2002ebb67a7918cf5841c6ca13c80278659d7678b9f785c"} Oct 30 18:43:21.398055 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:21.398022 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" Oct 30 18:43:22.449405 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:22.449370 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" event={"ID":"a6f75d54-cda6-4732-aedd-1250c4aa2c57","Type":"ContainerStarted","Data":"c1b732e506e99ee9b82e5d3be99ce01ffe083cf4022b0a0cc9e203a9dcd73ba1"} Oct 30 18:43:22.470156 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:22.470102 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f89dc687f-h9g5s" podStartSLOduration=1.264695052 podStartE2EDuration="4.470087414s" totalImagesPullingTime="3.205392362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:22.468751844 +0000 UTC m=+382.797163250" watchObservedRunningTime="2026-10-30 18:43:22.470087414 +0000 UTC m=+382.798498796" Oct 30 18:43:24.461564 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:24.461529 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" event={"ID":"a5f538d4-ba0c-4f64-953d-5f779027b94c","Type":"ContainerStarted","Data":"c297dc28d925035a6b85dc7cec5064dfe95169da046de9d96df488102ab242f7"} Oct 30 18:43:24.477964 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:24.477911 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-crll9" podStartSLOduration=1.80381426 podStartE2EDuration="7.477895163s" totalImagesPullingTime="5.674080903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:24.476592067 +0000 UTC m=+384.805003452" watchObservedRunningTime="2026-10-30 18:43:24.477895163 +0000 UTC m=+384.806306560" Oct 30 18:43:27.778972 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:43:27.778929 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:43:57.780758 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:43:57.780712 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:43:59.649122 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.649086 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:43:59.676861 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.676827 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:43:59.677025 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.676959 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:43:59.679018 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.678990 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:43:59.680675 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.680648 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 18:43:59.837424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.837392 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bb1241e0-96c3-480d-b527-cd2662e82ef2-config-file\") pod \"limitador-limitador-7c9b87d574-d9fgc\" (UID: \"bb1241e0-96c3-480d-b527-cd2662e82ef2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:43:59.837594 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.837439 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmq2x\" (UniqueName: \"kubernetes.io/projected/bb1241e0-96c3-480d-b527-cd2662e82ef2-kube-api-access-fmq2x\") pod \"limitador-limitador-7c9b87d574-d9fgc\" (UID: \"bb1241e0-96c3-480d-b527-cd2662e82ef2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:43:59.938191 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.938110 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmq2x\" (UniqueName: \"kubernetes.io/projected/bb1241e0-96c3-480d-b527-cd2662e82ef2-kube-api-access-fmq2x\") pod \"limitador-limitador-7c9b87d574-d9fgc\" (UID: \"bb1241e0-96c3-480d-b527-cd2662e82ef2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:43:59.938356 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.938204 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bb1241e0-96c3-480d-b527-cd2662e82ef2-config-file\") pod \"limitador-limitador-7c9b87d574-d9fgc\" (UID: \"bb1241e0-96c3-480d-b527-cd2662e82ef2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:43:59.940946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.940929 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 18:43:59.947218 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.947181 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmq2x\" (UniqueName: \"kubernetes.io/projected/bb1241e0-96c3-480d-b527-cd2662e82ef2-kube-api-access-fmq2x\") pod \"limitador-limitador-7c9b87d574-d9fgc\" (UID: \"bb1241e0-96c3-480d-b527-cd2662e82ef2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:43:59.949252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.949224 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bb1241e0-96c3-480d-b527-cd2662e82ef2-config-file\") pod \"limitador-limitador-7c9b87d574-d9fgc\" (UID: \"bb1241e0-96c3-480d-b527-cd2662e82ef2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:43:59.990641 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:43:59.990588 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:44:00.263590 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:00.263504 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:44:00.266409 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:44:00.266378 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1241e0_96c3_480d_b527_cd2662e82ef2.slice/crio-0697de790ae1edd8cd3e024c5d23e5241dd4aed9acc77b3fcd0c567ef81872b4 WatchSource:0}: Error finding container 0697de790ae1edd8cd3e024c5d23e5241dd4aed9acc77b3fcd0c567ef81872b4: Status 404 returned error can't find the container with id 0697de790ae1edd8cd3e024c5d23e5241dd4aed9acc77b3fcd0c567ef81872b4 Oct 30 18:44:00.624327 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:00.624235 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" event={"ID":"bb1241e0-96c3-480d-b527-cd2662e82ef2","Type":"ContainerStarted","Data":"0697de790ae1edd8cd3e024c5d23e5241dd4aed9acc77b3fcd0c567ef81872b4"} Oct 30 18:44:01.233508 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.233471 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5vlgx"] Oct 30 18:44:01.237301 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.237265 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" Oct 30 18:44:01.243828 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.243775 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5vlgx"] Oct 30 18:44:01.245376 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.245349 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-99gx9\"" Oct 30 18:44:01.353258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.353216 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2js\" (UniqueName: \"kubernetes.io/projected/5fac0c33-1c44-4cd0-8970-689be51997f5-kube-api-access-xn2js\") pod \"authorino-56c6c9ddd5-5vlgx\" (UID: \"5fac0c33-1c44-4cd0-8970-689be51997f5\") " pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" Oct 30 18:44:01.454961 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.454915 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2js\" (UniqueName: \"kubernetes.io/projected/5fac0c33-1c44-4cd0-8970-689be51997f5-kube-api-access-xn2js\") pod \"authorino-56c6c9ddd5-5vlgx\" (UID: \"5fac0c33-1c44-4cd0-8970-689be51997f5\") " pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" Oct 30 18:44:01.464398 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.464360 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2js\" (UniqueName: \"kubernetes.io/projected/5fac0c33-1c44-4cd0-8970-689be51997f5-kube-api-access-xn2js\") pod \"authorino-56c6c9ddd5-5vlgx\" (UID: \"5fac0c33-1c44-4cd0-8970-689be51997f5\") " pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" Oct 30 18:44:01.555937 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.555851 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" Oct 30 18:44:01.842555 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:01.840869 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5vlgx"] Oct 30 18:44:01.845277 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:44:01.844634 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fac0c33_1c44_4cd0_8970_689be51997f5.slice/crio-90b991489eeb51977bdd9eb6c42a1da318e27b0daaab794a1e73b0e3d68b3ff5 WatchSource:0}: Error finding container 90b991489eeb51977bdd9eb6c42a1da318e27b0daaab794a1e73b0e3d68b3ff5: Status 404 returned error can't find the container with id 90b991489eeb51977bdd9eb6c42a1da318e27b0daaab794a1e73b0e3d68b3ff5 Oct 30 18:44:02.636401 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:02.636358 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" event={"ID":"5fac0c33-1c44-4cd0-8970-689be51997f5","Type":"ContainerStarted","Data":"90b991489eeb51977bdd9eb6c42a1da318e27b0daaab794a1e73b0e3d68b3ff5"} Oct 30 18:44:03.647026 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:03.646980 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" event={"ID":"bb1241e0-96c3-480d-b527-cd2662e82ef2","Type":"ContainerStarted","Data":"768ebf246f380fb818999044d5de3f1e358fa4d8f602d0515797938d6413dd24"} Oct 30 18:44:03.647512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:03.647071 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:44:03.666751 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:03.666700 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" podStartSLOduration=2.073757701 podStartE2EDuration="4.666681543s" totalImagesPullingTime="2.592923842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:03.665435019 +0000 UTC m=+423.993846423" watchObservedRunningTime="2026-10-30 18:44:03.666681543 +0000 UTC m=+423.995092929" Oct 30 18:44:04.653679 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:04.653641 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" event={"ID":"5fac0c33-1c44-4cd0-8970-689be51997f5","Type":"ContainerStarted","Data":"1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7"} Oct 30 18:44:04.669777 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:04.669727 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" podStartSLOduration=1.7971451410000001 podStartE2EDuration="3.669709543s" totalImagesPullingTime="1.872564402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:04.668545702 +0000 UTC m=+424.996957100" watchObservedRunningTime="2026-10-30 18:44:04.669709543 +0000 UTC m=+424.998120925" Oct 30 18:44:14.117715 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.117678 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5"] Oct 30 18:44:14.121252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.121228 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:14.124917 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.124893 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:44:14.125771 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.125752 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:44:14.130010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.129981 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5"] Oct 30 18:44:14.133852 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.133836 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rkhfc\"" Oct 30 18:44:14.134654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.134632 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 18:44:14.146139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.146115 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zjvvq"] Oct 30 18:44:14.149694 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.149676 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:14.162686 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.162661 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ac4622-b2eb-41f1-86da-449e6b3b1170-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-z4zm5\" (UID: \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:14.162825 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.162696 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97mq\" (UniqueName: \"kubernetes.io/projected/523e3b72-74f3-4826-b00e-c0c7688620c4-kube-api-access-k97mq\") pod \"model-serving-api-75bd89fdfc-zjvvq\" (UID: \"523e3b72-74f3-4826-b00e-c0c7688620c4\") " pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:14.162893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.162851 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztt2r\" (UniqueName: \"kubernetes.io/projected/e5ac4622-b2eb-41f1-86da-449e6b3b1170-kube-api-access-ztt2r\") pod \"llmisvc-controller-manager-869d5dcc5d-z4zm5\" (UID: \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:14.162942 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.162897 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523e3b72-74f3-4826-b00e-c0c7688620c4-tls-certs\") pod \"model-serving-api-75bd89fdfc-zjvvq\" (UID: \"523e3b72-74f3-4826-b00e-c0c7688620c4\") " pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:14.168857 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.168825 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zjvvq"] Oct 30 18:44:14.188399 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.188371 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 18:44:14.188596 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.188414 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-8vsh5\"" Oct 30 18:44:14.264100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.264068 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ac4622-b2eb-41f1-86da-449e6b3b1170-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-z4zm5\" (UID: \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:14.264268 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.264115 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97mq\" (UniqueName: \"kubernetes.io/projected/523e3b72-74f3-4826-b00e-c0c7688620c4-kube-api-access-k97mq\") pod \"model-serving-api-75bd89fdfc-zjvvq\" (UID: \"523e3b72-74f3-4826-b00e-c0c7688620c4\") " pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:14.264268 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.264185 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztt2r\" (UniqueName: \"kubernetes.io/projected/e5ac4622-b2eb-41f1-86da-449e6b3b1170-kube-api-access-ztt2r\") pod \"llmisvc-controller-manager-869d5dcc5d-z4zm5\" (UID: \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:14.264268 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.264227 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523e3b72-74f3-4826-b00e-c0c7688620c4-tls-certs\") pod \"model-serving-api-75bd89fdfc-zjvvq\" (UID: \"523e3b72-74f3-4826-b00e-c0c7688620c4\") " pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:14.266915 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.266884 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ac4622-b2eb-41f1-86da-449e6b3b1170-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-z4zm5\" (UID: \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:14.267033 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.266884 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523e3b72-74f3-4826-b00e-c0c7688620c4-tls-certs\") pod \"model-serving-api-75bd89fdfc-zjvvq\" (UID: \"523e3b72-74f3-4826-b00e-c0c7688620c4\") " pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:14.283412 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.283385 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97mq\" (UniqueName: \"kubernetes.io/projected/523e3b72-74f3-4826-b00e-c0c7688620c4-kube-api-access-k97mq\") pod \"model-serving-api-75bd89fdfc-zjvvq\" (UID: \"523e3b72-74f3-4826-b00e-c0c7688620c4\") " pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:14.283564 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.283545 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztt2r\" (UniqueName: \"kubernetes.io/projected/e5ac4622-b2eb-41f1-86da-449e6b3b1170-kube-api-access-ztt2r\") pod \"llmisvc-controller-manager-869d5dcc5d-z4zm5\" (UID: \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:14.435718 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.435673 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:14.463165 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.463132 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:14.655240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.655203 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-d9fgc" Oct 30 18:44:14.715256 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.715234 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5"] Oct 30 18:44:14.718056 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:44:14.718028 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode5ac4622_b2eb_41f1_86da_449e6b3b1170.slice/crio-611d5ede50f72e9f2fea250ecff630bba1bbc52bbfd5d577c1c3afac547b3cdd WatchSource:0}: Error finding container 611d5ede50f72e9f2fea250ecff630bba1bbc52bbfd5d577c1c3afac547b3cdd: Status 404 returned error can't find the container with id 611d5ede50f72e9f2fea250ecff630bba1bbc52bbfd5d577c1c3afac547b3cdd Oct 30 18:44:14.758104 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:14.758068 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zjvvq"] Oct 30 18:44:14.761047 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:44:14.761023 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523e3b72_74f3_4826_b00e_c0c7688620c4.slice/crio-18fa21b1d1b70fd8e592ddd4e069241784350c47e2ee4bcb8d9bdf6a031cb688 WatchSource:0}: Error finding container 18fa21b1d1b70fd8e592ddd4e069241784350c47e2ee4bcb8d9bdf6a031cb688: Status 404 returned error can't find the container with id 18fa21b1d1b70fd8e592ddd4e069241784350c47e2ee4bcb8d9bdf6a031cb688 Oct 30 18:44:15.707535 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:15.707448 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" event={"ID":"e5ac4622-b2eb-41f1-86da-449e6b3b1170","Type":"ContainerStarted","Data":"611d5ede50f72e9f2fea250ecff630bba1bbc52bbfd5d577c1c3afac547b3cdd"} Oct 30 18:44:15.709542 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:15.709426 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" event={"ID":"523e3b72-74f3-4826-b00e-c0c7688620c4","Type":"ContainerStarted","Data":"18fa21b1d1b70fd8e592ddd4e069241784350c47e2ee4bcb8d9bdf6a031cb688"} Oct 30 18:44:17.722627 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:17.722570 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" event={"ID":"e5ac4622-b2eb-41f1-86da-449e6b3b1170","Type":"ContainerStarted","Data":"d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f"} Oct 30 18:44:17.723124 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:17.722664 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:17.724230 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:17.724207 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" event={"ID":"523e3b72-74f3-4826-b00e-c0c7688620c4","Type":"ContainerStarted","Data":"51d8fc05cf5da0cac9206029af0822c892f7b875bba694f39aac0f0d155920f4"} Oct 30 18:44:17.724361 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:17.724270 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:17.740441 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:17.740388 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" podStartSLOduration=1.4533018100000001 podStartE2EDuration="3.740373567s" totalImagesPullingTime="2.287071757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:17.739112868 +0000 UTC m=+438.067524254" watchObservedRunningTime="2026-10-30 18:44:17.740373567 +0000 UTC m=+438.068784948" Oct 30 18:44:17.756204 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:17.756149 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" podStartSLOduration=1.828016634 podStartE2EDuration="3.756134508s" totalImagesPullingTime="1.928117874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:17.755412449 +0000 UTC m=+438.083823835" watchObservedRunningTime="2026-10-30 18:44:17.756134508 +0000 UTC m=+438.084545892" Oct 30 18:44:27.782633 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:44:27.782571 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:44:28.733696 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:28.733667 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-zjvvq" Oct 30 18:44:48.731237 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:44:48.731138 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:44:57.780415 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:44:57.780373 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:45:13.184706 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.184670 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8547474b99-9pvxt"] Oct 30 18:45:13.188296 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.188275 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.194931 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.194902 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:45:13.200317 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.200288 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8547474b99-9pvxt"] Oct 30 18:45:13.201491 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.201462 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:45:13.202389 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.202372 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 18:45:13.202470 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.202375 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pd8db\"" Oct 30 18:45:13.262598 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.262560 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-maas-api-tls\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.262790 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.262626 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-metrics-certs\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.262790 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.262736 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzkd\" (UniqueName: \"kubernetes.io/projected/29561e69-cd6e-4101-b147-f05d258f45ed-kube-api-access-lpzkd\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.262872 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.262858 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29561e69-cd6e-4101-b147-f05d258f45ed-openshift-service-ca\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.295421 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.295388 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:45:13.364061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.364024 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29561e69-cd6e-4101-b147-f05d258f45ed-openshift-service-ca\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.364185 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.364082 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-maas-api-tls\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.364185 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.364109 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-metrics-certs\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.364185 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.364138 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzkd\" (UniqueName: \"kubernetes.io/projected/29561e69-cd6e-4101-b147-f05d258f45ed-kube-api-access-lpzkd\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.364887 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.364859 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29561e69-cd6e-4101-b147-f05d258f45ed-openshift-service-ca\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.366778 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.366757 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-maas-api-tls\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.366863 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.366809 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-metrics-certs\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.372492 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.372464 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzkd\" (UniqueName: \"kubernetes.io/projected/29561e69-cd6e-4101-b147-f05d258f45ed-kube-api-access-lpzkd\") pod \"maas-api-8547474b99-9pvxt\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.501554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.501478 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:13.778954 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.778872 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8547474b99-9pvxt"] Oct 30 18:45:13.782463 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:45:13.782437 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29561e69_cd6e_4101_b147_f05d258f45ed.slice/crio-a4a474dc2b8067a423ad033f878bf6679050888a69977e9ed6f903c84cfc1e3a WatchSource:0}: Error finding container a4a474dc2b8067a423ad033f878bf6679050888a69977e9ed6f903c84cfc1e3a: Status 404 returned error can't find the container with id a4a474dc2b8067a423ad033f878bf6679050888a69977e9ed6f903c84cfc1e3a Oct 30 18:45:13.907582 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.907545 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:45:13.986442 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:13.986402 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" event={"ID":"29561e69-cd6e-4101-b147-f05d258f45ed","Type":"ContainerStarted","Data":"a4a474dc2b8067a423ad033f878bf6679050888a69977e9ed6f903c84cfc1e3a"} Oct 30 18:45:18.008768 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:18.008647 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" event={"ID":"29561e69-cd6e-4101-b147-f05d258f45ed","Type":"ContainerStarted","Data":"13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355"} Oct 30 18:45:18.009173 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:18.008842 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:18.025570 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:18.025513 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" podStartSLOduration=1.140788306 podStartE2EDuration="5.0254997s" totalImagesPullingTime="3.884711394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:18.025061866 +0000 UTC m=+498.353473252" watchObservedRunningTime="2026-10-30 18:45:18.0254997 +0000 UTC m=+498.353911076" Oct 30 18:45:24.019546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:24.019517 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:24.882042 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:24.882005 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-wzs5x"] Oct 30 18:45:24.885836 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:24.885812 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:24.894165 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:24.894134 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-wzs5x"] Oct 30 18:45:24.919428 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:24.919395 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 18:45:24.992770 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:24.992733 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t262z\" (UniqueName: \"kubernetes.io/projected/04d5da0b-0680-4c70-9149-1ffcbffac507-kube-api-access-t262z\") pod \"authorino-6889cbd8c4-wzs5x\" (UID: \"04d5da0b-0680-4c70-9149-1ffcbffac507\") " pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:24.992991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:24.992919 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04d5da0b-0680-4c70-9149-1ffcbffac507-tls-cert\") pod \"authorino-6889cbd8c4-wzs5x\" (UID: \"04d5da0b-0680-4c70-9149-1ffcbffac507\") " pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:25.093930 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.093890 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t262z\" (UniqueName: \"kubernetes.io/projected/04d5da0b-0680-4c70-9149-1ffcbffac507-kube-api-access-t262z\") pod \"authorino-6889cbd8c4-wzs5x\" (UID: \"04d5da0b-0680-4c70-9149-1ffcbffac507\") " pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:25.094351 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.094007 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04d5da0b-0680-4c70-9149-1ffcbffac507-tls-cert\") pod \"authorino-6889cbd8c4-wzs5x\" (UID: \"04d5da0b-0680-4c70-9149-1ffcbffac507\") " pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:25.096648 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.096627 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04d5da0b-0680-4c70-9149-1ffcbffac507-tls-cert\") pod \"authorino-6889cbd8c4-wzs5x\" (UID: \"04d5da0b-0680-4c70-9149-1ffcbffac507\") " pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:25.103347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.103314 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t262z\" (UniqueName: \"kubernetes.io/projected/04d5da0b-0680-4c70-9149-1ffcbffac507-kube-api-access-t262z\") pod \"authorino-6889cbd8c4-wzs5x\" (UID: \"04d5da0b-0680-4c70-9149-1ffcbffac507\") " pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:25.198525 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.198480 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:25.242333 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.242295 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-wzs5x"] Oct 30 18:45:25.271408 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.271376 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6867c46776-hlcsj"] Oct 30 18:45:25.275854 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.275825 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:45:25.350516 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.350478 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6867c46776-hlcsj"] Oct 30 18:45:25.397151 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.397114 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nhl\" (UniqueName: \"kubernetes.io/projected/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-kube-api-access-v4nhl\") pod \"authorino-6867c46776-hlcsj\" (UID: \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\") " pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:45:25.397151 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.397152 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-tls-cert\") pod \"authorino-6867c46776-hlcsj\" (UID: \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\") " pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:45:25.475931 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.475846 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-wzs5x"] Oct 30 18:45:25.478958 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:45:25.478917 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04d5da0b_0680_4c70_9149_1ffcbffac507.slice/crio-9a5ba3220e1ad49abc2dedeb430847392f6b9bbf6c619e35fda20d68d87c484d WatchSource:0}: Error finding container 9a5ba3220e1ad49abc2dedeb430847392f6b9bbf6c619e35fda20d68d87c484d: Status 404 returned error can't find the container with id 9a5ba3220e1ad49abc2dedeb430847392f6b9bbf6c619e35fda20d68d87c484d Oct 30 18:45:25.498023 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.497991 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nhl\" (UniqueName: \"kubernetes.io/projected/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-kube-api-access-v4nhl\") pod \"authorino-6867c46776-hlcsj\" (UID: \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\") " pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:45:25.498023 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.498028 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-tls-cert\") pod \"authorino-6867c46776-hlcsj\" (UID: \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\") " pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:45:25.500695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.500673 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-tls-cert\") pod \"authorino-6867c46776-hlcsj\" (UID: \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\") " pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:45:25.509662 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.509636 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nhl\" (UniqueName: \"kubernetes.io/projected/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-kube-api-access-v4nhl\") pod \"authorino-6867c46776-hlcsj\" (UID: \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\") " pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:45:25.589512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.589473 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:45:25.863682 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:25.863514 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6867c46776-hlcsj"] Oct 30 18:45:25.866350 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:45:25.866319 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2eac99e_bcdc_4278_b0d7_5dadefd2ed1c.slice/crio-fdb8ec938111023fd194fc52cf3bed6694a165b5d3f10300df0f66e6715c3deb WatchSource:0}: Error finding container fdb8ec938111023fd194fc52cf3bed6694a165b5d3f10300df0f66e6715c3deb: Status 404 returned error can't find the container with id fdb8ec938111023fd194fc52cf3bed6694a165b5d3f10300df0f66e6715c3deb Oct 30 18:45:26.046513 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:26.046470 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" event={"ID":"04d5da0b-0680-4c70-9149-1ffcbffac507","Type":"ContainerStarted","Data":"9a5ba3220e1ad49abc2dedeb430847392f6b9bbf6c619e35fda20d68d87c484d"} Oct 30 18:45:26.047802 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:26.047779 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6867c46776-hlcsj" event={"ID":"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c","Type":"ContainerStarted","Data":"fdb8ec938111023fd194fc52cf3bed6694a165b5d3f10300df0f66e6715c3deb"} Oct 30 18:45:27.056488 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.056446 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" event={"ID":"04d5da0b-0680-4c70-9149-1ffcbffac507","Type":"ContainerStarted","Data":"0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5"} Oct 30 18:45:27.057070 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.057036 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" podUID="04d5da0b-0680-4c70-9149-1ffcbffac507" containerName="authorino" containerID="cri-o://0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5" gracePeriod=30 Oct 30 18:45:27.058442 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.058408 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6867c46776-hlcsj" event={"ID":"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c","Type":"ContainerStarted","Data":"7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946"} Oct 30 18:45:27.090100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.090031 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6867c46776-hlcsj" podStartSLOduration=1.509179691 podStartE2EDuration="2.090012141s" totalImagesPullingTime="580.83245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:27.089367759 +0000 UTC m=+507.417779146" watchObservedRunningTime="2026-10-30 18:45:27.090012141 +0000 UTC m=+507.418423527" Oct 30 18:45:27.091036 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.090993 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" podStartSLOduration=2.21499149 podStartE2EDuration="3.090982462s" totalImagesPullingTime="875.990972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:27.072794412 +0000 UTC m=+507.401205798" watchObservedRunningTime="2026-10-30 18:45:27.090982462 +0000 UTC m=+507.419393846" Oct 30 18:45:27.117331 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.117260 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5vlgx"] Oct 30 18:45:27.117592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.117563 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" podUID="5fac0c33-1c44-4cd0-8970-689be51997f5" containerName="authorino" containerID="cri-o://1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7" gracePeriod=30 Oct 30 18:45:27.393988 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.393961 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:27.397173 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.397150 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" Oct 30 18:45:27.420279 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.420249 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d5da0b-0680-4c70-9149-1ffcbffac507-tls-cert\" (UniqueName: \"kubernetes.io/secret/04d5da0b-0680-4c70-9149-1ffcbffac507-tls-cert\") pod \"04d5da0b-0680-4c70-9149-1ffcbffac507\" (UID: \"04d5da0b-0680-4c70-9149-1ffcbffac507\") " Oct 30 18:45:27.420459 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.420293 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fac0c33-1c44-4cd0-8970-689be51997f5-kube-api-access-xn2js\" (UniqueName: \"kubernetes.io/projected/5fac0c33-1c44-4cd0-8970-689be51997f5-kube-api-access-xn2js\") pod \"5fac0c33-1c44-4cd0-8970-689be51997f5\" (UID: \"5fac0c33-1c44-4cd0-8970-689be51997f5\") " Oct 30 18:45:27.420459 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.420321 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d5da0b-0680-4c70-9149-1ffcbffac507-kube-api-access-t262z\" (UniqueName: \"kubernetes.io/projected/04d5da0b-0680-4c70-9149-1ffcbffac507-kube-api-access-t262z\") pod \"04d5da0b-0680-4c70-9149-1ffcbffac507\" (UID: \"04d5da0b-0680-4c70-9149-1ffcbffac507\") " Oct 30 18:45:27.422679 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.422647 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fac0c33-1c44-4cd0-8970-689be51997f5-kube-api-access-xn2js" pod "5fac0c33-1c44-4cd0-8970-689be51997f5" (UID: "5fac0c33-1c44-4cd0-8970-689be51997f5"). InnerVolumeSpecName "kube-api-access-xn2js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:27.422791 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.422679 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d5da0b-0680-4c70-9149-1ffcbffac507-kube-api-access-t262z" pod "04d5da0b-0680-4c70-9149-1ffcbffac507" (UID: "04d5da0b-0680-4c70-9149-1ffcbffac507"). InnerVolumeSpecName "kube-api-access-t262z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:27.432943 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.432896 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d5da0b-0680-4c70-9149-1ffcbffac507-tls-cert" pod "04d5da0b-0680-4c70-9149-1ffcbffac507" (UID: "04d5da0b-0680-4c70-9149-1ffcbffac507"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:27.522101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.522060 5959 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/04d5da0b-0680-4c70-9149-1ffcbffac507-tls-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:27.522101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.522096 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2js\" (UniqueName: \"kubernetes.io/projected/5fac0c33-1c44-4cd0-8970-689be51997f5-kube-api-access-xn2js\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:27.522101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:27.522108 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t262z\" (UniqueName: \"kubernetes.io/projected/04d5da0b-0680-4c70-9149-1ffcbffac507-kube-api-access-t262z\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:27.781584 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:45:27.781543 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:45:28.064338 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.064225 5959 generic.go:356] "Generic (PLEG): container finished" podID="04d5da0b-0680-4c70-9149-1ffcbffac507" containerID="0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5" exitCode=0 Oct 30 18:45:28.064338 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.064275 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" Oct 30 18:45:28.064338 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.064311 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" event={"ID":"04d5da0b-0680-4c70-9149-1ffcbffac507","Type":"ContainerDied","Data":"0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5"} Oct 30 18:45:28.064908 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.064352 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-wzs5x" event={"ID":"04d5da0b-0680-4c70-9149-1ffcbffac507","Type":"ContainerDied","Data":"9a5ba3220e1ad49abc2dedeb430847392f6b9bbf6c619e35fda20d68d87c484d"} Oct 30 18:45:28.064908 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.064376 5959 scope.go:122] "RemoveContainer" containerID="0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5" Oct 30 18:45:28.065731 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.065712 5959 generic.go:356] "Generic (PLEG): container finished" podID="5fac0c33-1c44-4cd0-8970-689be51997f5" containerID="1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7" exitCode=0 Oct 30 18:45:28.065831 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.065756 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" Oct 30 18:45:28.065831 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.065806 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" event={"ID":"5fac0c33-1c44-4cd0-8970-689be51997f5","Type":"ContainerDied","Data":"1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7"} Oct 30 18:45:28.065932 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.065849 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5vlgx" event={"ID":"5fac0c33-1c44-4cd0-8970-689be51997f5","Type":"ContainerDied","Data":"90b991489eeb51977bdd9eb6c42a1da318e27b0daaab794a1e73b0e3d68b3ff5"} Oct 30 18:45:28.073960 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.073939 5959 scope.go:122] "RemoveContainer" containerID="0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5" Oct 30 18:45:28.074249 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:45:28.074227 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5\": container with ID starting with 0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5 not found: ID does not exist" containerID="0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5" Oct 30 18:45:28.074307 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.074262 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5"} err="failed to get container status \"0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5\": rpc error: code = NotFound desc = could not find container \"0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5\": container with ID starting with 0f97bf4098b040ab374579bdc46d7825599a7ae0e666eb06ace121af4405ade5 not found: ID does not exist" Oct 30 18:45:28.074307 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.074287 5959 scope.go:122] "RemoveContainer" containerID="1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7" Oct 30 18:45:28.079986 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.079965 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw"] Oct 30 18:45:28.080380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.080368 5959 cpu_manager.go:414] "containerMap: removing container" podUID="04d5da0b-0680-4c70-9149-1ffcbffac507" containerName="authorino" Oct 30 18:45:28.080423 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.080383 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d5da0b-0680-4c70-9149-1ffcbffac507" containerName="authorino" Oct 30 18:45:28.080423 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.080400 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5fac0c33-1c44-4cd0-8970-689be51997f5" containerName="authorino" Oct 30 18:45:28.080423 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.080405 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fac0c33-1c44-4cd0-8970-689be51997f5" containerName="authorino" Oct 30 18:45:28.080512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.080463 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d5da0b-0680-4c70-9149-1ffcbffac507" containerName="authorino" Oct 30 18:45:28.080512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.080473 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fac0c33-1c44-4cd0-8970-689be51997f5" containerName="authorino" Oct 30 18:45:28.083311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.083294 5959 scope.go:122] "RemoveContainer" containerID="1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7" Oct 30 18:45:28.083637 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:45:28.083590 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7\": container with ID starting with 1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7 not found: ID does not exist" containerID="1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7" Oct 30 18:45:28.083751 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.083645 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7"} err="failed to get container status \"1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7\": rpc error: code = NotFound desc = could not find container \"1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7\": container with ID starting with 1cd47912f1c898945a4f010460f14ea1ec9b243f7e03120000afd3b6838ab2d7 not found: ID does not exist" Oct 30 18:45:28.084824 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.084810 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:28.097826 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.097798 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-wzs5x"] Oct 30 18:45:28.100885 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.100803 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw"] Oct 30 18:45:28.102014 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.101988 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-wzs5x"] Oct 30 18:45:28.119193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.119152 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5vlgx"] Oct 30 18:45:28.122549 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.122521 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5vlgx"] Oct 30 18:45:28.127438 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.127408 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmr6\" (UniqueName: \"kubernetes.io/projected/1e5b9831-369a-4cbd-b808-0e3dca949a09-kube-api-access-stmr6\") pod \"llmisvc-controller-manager-6466c554d8-7gnzw\" (UID: \"1e5b9831-369a-4cbd-b808-0e3dca949a09\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:28.127564 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.127517 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e5b9831-369a-4cbd-b808-0e3dca949a09-cert\") pod \"llmisvc-controller-manager-6466c554d8-7gnzw\" (UID: \"1e5b9831-369a-4cbd-b808-0e3dca949a09\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:28.228500 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.228459 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmr6\" (UniqueName: \"kubernetes.io/projected/1e5b9831-369a-4cbd-b808-0e3dca949a09-kube-api-access-stmr6\") pod \"llmisvc-controller-manager-6466c554d8-7gnzw\" (UID: \"1e5b9831-369a-4cbd-b808-0e3dca949a09\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:28.228686 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.228542 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e5b9831-369a-4cbd-b808-0e3dca949a09-cert\") pod \"llmisvc-controller-manager-6466c554d8-7gnzw\" (UID: \"1e5b9831-369a-4cbd-b808-0e3dca949a09\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:28.231096 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.231076 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e5b9831-369a-4cbd-b808-0e3dca949a09-cert\") pod \"llmisvc-controller-manager-6466c554d8-7gnzw\" (UID: \"1e5b9831-369a-4cbd-b808-0e3dca949a09\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:28.237794 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.237768 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmr6\" (UniqueName: \"kubernetes.io/projected/1e5b9831-369a-4cbd-b808-0e3dca949a09-kube-api-access-stmr6\") pod \"llmisvc-controller-manager-6466c554d8-7gnzw\" (UID: \"1e5b9831-369a-4cbd-b808-0e3dca949a09\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:28.398384 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.398345 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:28.686341 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:28.686246 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw"] Oct 30 18:45:28.690080 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:45:28.690041 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5b9831_369a_4cbd_b808_0e3dca949a09.slice/crio-c75c7faef7e56b666112923d24fc0a1f55b9c86ef7866c9f1ba7f0c7c0590281 WatchSource:0}: Error finding container c75c7faef7e56b666112923d24fc0a1f55b9c86ef7866c9f1ba7f0c7c0590281: Status 404 returned error can't find the container with id c75c7faef7e56b666112923d24fc0a1f55b9c86ef7866c9f1ba7f0c7c0590281 Oct 30 18:45:29.075462 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:29.075422 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" event={"ID":"1e5b9831-369a-4cbd-b808-0e3dca949a09","Type":"ContainerStarted","Data":"c75c7faef7e56b666112923d24fc0a1f55b9c86ef7866c9f1ba7f0c7c0590281"} Oct 30 18:45:29.796164 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:29.796134 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d5da0b-0680-4c70-9149-1ffcbffac507" path="/var/lib/kubelet/pods/04d5da0b-0680-4c70-9149-1ffcbffac507/volumes" Oct 30 18:45:29.796470 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:29.796458 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fac0c33-1c44-4cd0-8970-689be51997f5" path="/var/lib/kubelet/pods/5fac0c33-1c44-4cd0-8970-689be51997f5/volumes" Oct 30 18:45:30.081287 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:30.081193 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" event={"ID":"1e5b9831-369a-4cbd-b808-0e3dca949a09","Type":"ContainerStarted","Data":"d7c6860f12977fdccd1da5c1ab98dddf382db08d1daa05fda5a016f7ce521686"} Oct 30 18:45:30.081776 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:30.081346 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:45:30.099670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:30.099599 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" podStartSLOduration=1.749672453 podStartE2EDuration="2.099582297s" totalImagesPullingTime="349.909844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:30.098445726 +0000 UTC m=+510.426857112" watchObservedRunningTime="2026-10-30 18:45:30.099582297 +0000 UTC m=+510.427993687" Oct 30 18:45:34.765869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:34.765833 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8547474b99-9pvxt"] Oct 30 18:45:34.766290 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:34.766034 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" podUID="29561e69-cd6e-4101-b147-f05d258f45ed" containerName="maas-api" containerID="cri-o://13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355" gracePeriod=30 Oct 30 18:45:35.013022 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.012995 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:35.096139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.096050 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29561e69-cd6e-4101-b147-f05d258f45ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29561e69-cd6e-4101-b147-f05d258f45ed-openshift-service-ca\") pod \"29561e69-cd6e-4101-b147-f05d258f45ed\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " Oct 30 18:45:35.096139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.096101 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-maas-api-tls\") pod \"29561e69-cd6e-4101-b147-f05d258f45ed\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " Oct 30 18:45:35.096353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.096215 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-metrics-certs\") pod \"29561e69-cd6e-4101-b147-f05d258f45ed\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " Oct 30 18:45:35.096353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.096248 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29561e69-cd6e-4101-b147-f05d258f45ed-kube-api-access-lpzkd\" (UniqueName: \"kubernetes.io/projected/29561e69-cd6e-4101-b147-f05d258f45ed-kube-api-access-lpzkd\") pod \"29561e69-cd6e-4101-b147-f05d258f45ed\" (UID: \"29561e69-cd6e-4101-b147-f05d258f45ed\") " Oct 30 18:45:35.096503 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.096477 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29561e69-cd6e-4101-b147-f05d258f45ed-openshift-service-ca" pod "29561e69-cd6e-4101-b147-f05d258f45ed" (UID: "29561e69-cd6e-4101-b147-f05d258f45ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:35.098593 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.098564 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-maas-api-tls" pod "29561e69-cd6e-4101-b147-f05d258f45ed" (UID: "29561e69-cd6e-4101-b147-f05d258f45ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:35.098593 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.098583 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-metrics-certs" pod "29561e69-cd6e-4101-b147-f05d258f45ed" (UID: "29561e69-cd6e-4101-b147-f05d258f45ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:35.098769 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.098619 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29561e69-cd6e-4101-b147-f05d258f45ed-kube-api-access-lpzkd" pod "29561e69-cd6e-4101-b147-f05d258f45ed" (UID: "29561e69-cd6e-4101-b147-f05d258f45ed"). InnerVolumeSpecName "kube-api-access-lpzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:35.106137 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.106108 5959 generic.go:356] "Generic (PLEG): container finished" podID="29561e69-cd6e-4101-b147-f05d258f45ed" containerID="13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355" exitCode=0 Oct 30 18:45:35.106215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.106157 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" Oct 30 18:45:35.106215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.106183 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" event={"ID":"29561e69-cd6e-4101-b147-f05d258f45ed","Type":"ContainerDied","Data":"13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355"} Oct 30 18:45:35.106305 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.106238 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8547474b99-9pvxt" event={"ID":"29561e69-cd6e-4101-b147-f05d258f45ed","Type":"ContainerDied","Data":"a4a474dc2b8067a423ad033f878bf6679050888a69977e9ed6f903c84cfc1e3a"} Oct 30 18:45:35.106305 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.106258 5959 scope.go:122] "RemoveContainer" containerID="13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355" Oct 30 18:45:35.123804 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.123781 5959 scope.go:122] "RemoveContainer" containerID="13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355" Oct 30 18:45:35.124092 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:45:35.124073 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355\": container with ID starting with 13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355 not found: ID does not exist" containerID="13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355" Oct 30 18:45:35.124142 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.124102 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355"} err="failed to get container status \"13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355\": rpc error: code = NotFound desc = could not find container \"13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355\": container with ID starting with 13ea88319a2e50db79ad2dd0985e869640466730d0b35d3eb0a87ecf4e44b355 not found: ID does not exist" Oct 30 18:45:35.131030 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.131000 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8547474b99-9pvxt"] Oct 30 18:45:35.135543 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.135522 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8547474b99-9pvxt"] Oct 30 18:45:35.197630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.197568 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:35.197630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.197599 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpzkd\" (UniqueName: \"kubernetes.io/projected/29561e69-cd6e-4101-b147-f05d258f45ed-kube-api-access-lpzkd\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:35.197630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.197636 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29561e69-cd6e-4101-b147-f05d258f45ed-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:35.197867 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.197645 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29561e69-cd6e-4101-b147-f05d258f45ed-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:35.795666 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:45:35.795631 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29561e69-cd6e-4101-b147-f05d258f45ed" path="/var/lib/kubelet/pods/29561e69-cd6e-4101-b147-f05d258f45ed/volumes" Oct 30 18:45:57.778311 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:45:57.778274 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:46:01.088593 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.088562 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7gnzw" Oct 30 18:46:01.150332 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.150294 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5"] Oct 30 18:46:01.150706 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.150678 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" podUID="e5ac4622-b2eb-41f1-86da-449e6b3b1170" containerName="manager" containerID="cri-o://d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f" gracePeriod=30 Oct 30 18:46:01.399282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.399255 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:46:01.548341 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.548308 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5ac4622-b2eb-41f1-86da-449e6b3b1170-cert\" (UniqueName: \"kubernetes.io/secret/e5ac4622-b2eb-41f1-86da-449e6b3b1170-cert\") pod \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\" (UID: \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\") " Oct 30 18:46:01.548504 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.548449 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ac4622-b2eb-41f1-86da-449e6b3b1170-kube-api-access-ztt2r\" (UniqueName: \"kubernetes.io/projected/e5ac4622-b2eb-41f1-86da-449e6b3b1170-kube-api-access-ztt2r\") pod \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\" (UID: \"e5ac4622-b2eb-41f1-86da-449e6b3b1170\") " Oct 30 18:46:01.550788 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.550756 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ac4622-b2eb-41f1-86da-449e6b3b1170-cert" pod "e5ac4622-b2eb-41f1-86da-449e6b3b1170" (UID: "e5ac4622-b2eb-41f1-86da-449e6b3b1170"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:01.550788 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.550778 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ac4622-b2eb-41f1-86da-449e6b3b1170-kube-api-access-ztt2r" pod "e5ac4622-b2eb-41f1-86da-449e6b3b1170" (UID: "e5ac4622-b2eb-41f1-86da-449e6b3b1170"). InnerVolumeSpecName "kube-api-access-ztt2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:01.649510 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.649485 5959 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ac4622-b2eb-41f1-86da-449e6b3b1170-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:01.649510 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:01.649509 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztt2r\" (UniqueName: \"kubernetes.io/projected/e5ac4622-b2eb-41f1-86da-449e6b3b1170-kube-api-access-ztt2r\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:02.231319 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.231279 5959 generic.go:356] "Generic (PLEG): container finished" podID="e5ac4622-b2eb-41f1-86da-449e6b3b1170" containerID="d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f" exitCode=0 Oct 30 18:46:02.231723 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.231354 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" event={"ID":"e5ac4622-b2eb-41f1-86da-449e6b3b1170","Type":"ContainerDied","Data":"d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f"} Oct 30 18:46:02.231723 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.231406 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" event={"ID":"e5ac4622-b2eb-41f1-86da-449e6b3b1170","Type":"ContainerDied","Data":"611d5ede50f72e9f2fea250ecff630bba1bbc52bbfd5d577c1c3afac547b3cdd"} Oct 30 18:46:02.231723 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.231435 5959 scope.go:122] "RemoveContainer" containerID="d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f" Oct 30 18:46:02.231723 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.231592 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5" Oct 30 18:46:02.242863 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.242844 5959 scope.go:122] "RemoveContainer" containerID="d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f" Oct 30 18:46:02.243169 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:46:02.243136 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f\": container with ID starting with d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f not found: ID does not exist" containerID="d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f" Oct 30 18:46:02.243312 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.243176 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f"} err="failed to get container status \"d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f\": rpc error: code = NotFound desc = could not find container \"d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f\": container with ID starting with d3f136c497b9929c9c3d015e9853c948e4c282c52182acbb6dea9c7cae85643f not found: ID does not exist" Oct 30 18:46:02.249044 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.249026 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5"] Oct 30 18:46:02.258373 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:02.258352 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-z4zm5"] Oct 30 18:46:03.795598 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:03.795567 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ac4622-b2eb-41f1-86da-449e6b3b1170" path="/var/lib/kubelet/pods/e5ac4622-b2eb-41f1-86da-449e6b3b1170/volumes" Oct 30 18:46:27.778654 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:46:27.778549 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:46:35.767774 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:35.767741 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:46:38.753463 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:38.753428 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:46:41.948182 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:41.948145 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:46:46.860173 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:46.860130 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:46:56.809539 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.809499 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5"] Oct 30 18:46:56.810123 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.810100 5959 cpu_manager.go:414] "containerMap: removing container" podUID="29561e69-cd6e-4101-b147-f05d258f45ed" containerName="maas-api" Oct 30 18:46:56.810220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.810127 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29561e69-cd6e-4101-b147-f05d258f45ed" containerName="maas-api" Oct 30 18:46:56.810220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.810150 5959 cpu_manager.go:414] "containerMap: removing container" podUID="e5ac4622-b2eb-41f1-86da-449e6b3b1170" containerName="manager" Oct 30 18:46:56.810220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.810158 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ac4622-b2eb-41f1-86da-449e6b3b1170" containerName="manager" Oct 30 18:46:56.810334 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.810264 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="29561e69-cd6e-4101-b147-f05d258f45ed" containerName="maas-api" Oct 30 18:46:56.810334 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.810276 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ac4622-b2eb-41f1-86da-449e6b3b1170" containerName="manager" Oct 30 18:46:56.813417 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.813394 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.829651 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.829631 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:46:56.831729 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.831702 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5"] Oct 30 18:46:56.880624 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.880578 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.881061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.880637 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfm89\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-kube-api-access-pfm89\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.881061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.880717 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.881061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.880762 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.881061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.880785 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.881061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.880880 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.881061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.880907 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.881061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.880936 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.881061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.881049 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.904098 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.904070 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sj2rm\"" Oct 30 18:46:56.982380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982346 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982573 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982398 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982573 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982517 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982573 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982560 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982670 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982741 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982769 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfm89\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-kube-api-access-pfm89\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982805 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982808 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.982898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982853 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.983254 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.982981 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.983254 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.983042 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.983254 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.983213 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.983622 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.983584 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.985282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.985261 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.985667 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.985649 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.991155 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.991127 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:56.991267 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:56.991212 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfm89\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-kube-api-access-pfm89\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-8hjs5\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:57.129078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:57.128999 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:46:57.413625 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:57.413585 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5"] Oct 30 18:46:57.416489 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:46:57.416453 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c7c1c0_16bf_4637_ae86_99e9a19dfd8a.slice/crio-1983897576c3f2dcbb479c3fbccae2a4442623afd8621248c3964ee2c27a80fb WatchSource:0}: Error finding container 1983897576c3f2dcbb479c3fbccae2a4442623afd8621248c3964ee2c27a80fb: Status 404 returned error can't find the container with id 1983897576c3f2dcbb479c3fbccae2a4442623afd8621248c3964ee2c27a80fb Oct 30 18:46:57.499507 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:46:57.499466 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" event={"ID":"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a","Type":"ContainerStarted","Data":"1983897576c3f2dcbb479c3fbccae2a4442623afd8621248c3964ee2c27a80fb"} Oct 30 18:46:57.780695 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:46:57.780593 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:47:00.765877 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:00.765848 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6867c46776-hlcsj"] Oct 30 18:47:00.766255 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:00.766065 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6867c46776-hlcsj" podUID="b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c" containerName="authorino" containerID="cri-o://7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946" gracePeriod=30 Oct 30 18:47:00.871460 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:00.871427 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:00.871554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:00.871510 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:00.871554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:00.871545 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:01.016554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.016496 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:47:01.122008 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.121979 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-kube-api-access-v4nhl\" (UniqueName: \"kubernetes.io/projected/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-kube-api-access-v4nhl\") pod \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\" (UID: \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\") " Oct 30 18:47:01.122179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.122020 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-tls-cert\" (UniqueName: \"kubernetes.io/secret/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-tls-cert\") pod \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\" (UID: \"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c\") " Oct 30 18:47:01.124263 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.124230 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-kube-api-access-v4nhl" pod "b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c" (UID: "b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c"). InnerVolumeSpecName "kube-api-access-v4nhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:01.133893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.133869 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-tls-cert" pod "b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c" (UID: "b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:01.223380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.223352 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4nhl\" (UniqueName: \"kubernetes.io/projected/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-kube-api-access-v4nhl\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:01.223380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.223377 5959 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c-tls-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:01.525253 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.525215 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" event={"ID":"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a","Type":"ContainerStarted","Data":"712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9"} Oct 30 18:47:01.526530 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.526505 5959 generic.go:356] "Generic (PLEG): container finished" podID="b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c" containerID="7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946" exitCode=0 Oct 30 18:47:01.526673 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.526540 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6867c46776-hlcsj" event={"ID":"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c","Type":"ContainerDied","Data":"7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946"} Oct 30 18:47:01.526673 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.526543 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6867c46776-hlcsj" Oct 30 18:47:01.526673 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.526572 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6867c46776-hlcsj" event={"ID":"b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c","Type":"ContainerDied","Data":"fdb8ec938111023fd194fc52cf3bed6694a165b5d3f10300df0f66e6715c3deb"} Oct 30 18:47:01.526673 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.526596 5959 scope.go:122] "RemoveContainer" containerID="7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946" Oct 30 18:47:01.535888 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.535869 5959 scope.go:122] "RemoveContainer" containerID="7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946" Oct 30 18:47:01.536145 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:47:01.536127 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946\": container with ID starting with 7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946 not found: ID does not exist" containerID="7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946" Oct 30 18:47:01.536197 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.536152 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946"} err="failed to get container status \"7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946\": rpc error: code = NotFound desc = could not find container \"7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946\": container with ID starting with 7a5d9ddd9440c55cebf4075899d0606bd950fe53494f06b1158bd77ba2e45946 not found: ID does not exist" Oct 30 18:47:01.548280 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.548227 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podStartSLOduration=2.096607634 podStartE2EDuration="5.548211071s" totalImagesPullingTime="3.451603437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:01.547883519 +0000 UTC m=+601.876294902" watchObservedRunningTime="2026-10-30 18:47:01.548211071 +0000 UTC m=+601.876622457" Oct 30 18:47:01.567751 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.567728 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6867c46776-hlcsj"] Oct 30 18:47:01.571642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.571595 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6867c46776-hlcsj"] Oct 30 18:47:01.794776 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:01.794701 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c" path="/var/lib/kubelet/pods/b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c/volumes" Oct 30 18:47:02.130021 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:02.129932 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:47:02.131347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:02.131321 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:47:02.131405 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:02.131383 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:47:03.129583 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:03.129546 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:47:03.129968 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:03.129628 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:47:04.130219 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:04.130179 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:47:04.130599 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:04.130251 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:47:05.129687 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:05.129650 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:47:05.129871 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:05.129706 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:47:06.130190 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:06.130151 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:47:06.130655 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:06.130211 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:47:07.129566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:07.129529 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:47:07.129836 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:07.129629 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:47:08.129511 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:08.129473 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:47:08.130034 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:08.129542 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:47:09.133896 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:09.133864 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:47:09.134365 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:09.134167 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:47:09.134808 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:09.134790 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 18:47:27.779039 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:47:27.779003 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:47:35.104111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.104072 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k"] Oct 30 18:47:35.104758 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.104741 5959 cpu_manager.go:414] "containerMap: removing container" podUID="b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c" containerName="authorino" Oct 30 18:47:35.104815 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.104761 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c" containerName="authorino" Oct 30 18:47:35.104880 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.104869 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2eac99e-bcdc-4278-b0d7-5dadefd2ed1c" containerName="authorino" Oct 30 18:47:35.114175 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.114148 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.119830 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.119804 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ee87bf29-openshift-default-dockercfg-d9js5\"" Oct 30 18:47:35.121375 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.120469 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k"] Oct 30 18:47:35.254594 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.254564 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-certs\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.254594 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.254621 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istiod-ca-cert\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.254842 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.254781 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-envoy\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.254842 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.254819 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-podinfo\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.254842 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.254840 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-token\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.255026 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.255007 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-credential-socket\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.255087 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.255052 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-socket\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.255202 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.255176 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-data\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.255253 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.255234 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bh4\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-kube-api-access-56bh4\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.356802 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.356706 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-credential-socket\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.356802 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.356773 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-socket\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357024 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.356832 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-data\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357024 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.356860 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bh4\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-kube-api-access-56bh4\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357024 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.356925 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-certs\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357231 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357108 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-credential-socket\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357231 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357147 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-socket\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357231 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357157 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istiod-ca-cert\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357396 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357231 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-envoy\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357396 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357274 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-podinfo\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357396 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357320 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-token\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357552 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357490 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-certs\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357552 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357527 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-data\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.357792 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.357758 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istiod-ca-cert\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.360045 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.360019 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-podinfo\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.360181 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.360160 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-envoy\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.365391 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.365362 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bh4\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-kube-api-access-56bh4\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.365886 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.365868 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-token\") pod \"e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.433469 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.433440 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:35.711453 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.711424 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k"] Oct 30 18:47:35.714400 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:47:35.714364 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6d9f29_13d9_4411_9b29_d3b4c56221e3.slice/crio-fdb44facc68dd09fe5374aaf33984c8e0998b8c9735160dc662d7f12b6ae44be WatchSource:0}: Error finding container fdb44facc68dd09fe5374aaf33984c8e0998b8c9735160dc662d7f12b6ae44be: Status 404 returned error can't find the container with id fdb44facc68dd09fe5374aaf33984c8e0998b8c9735160dc662d7f12b6ae44be Oct 30 18:47:35.716712 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.716681 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:35.716789 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.716748 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:35.716789 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:35.716778 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:36.690590 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:36.690554 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" event={"ID":"ac6d9f29-13d9-4411-9b29-d3b4c56221e3","Type":"ContainerStarted","Data":"654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c"} Oct 30 18:47:36.690995 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:36.690641 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" event={"ID":"ac6d9f29-13d9-4411-9b29-d3b4c56221e3","Type":"ContainerStarted","Data":"fdb44facc68dd09fe5374aaf33984c8e0998b8c9735160dc662d7f12b6ae44be"} Oct 30 18:47:36.715828 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:36.715746 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" podStartSLOduration=1.715733534 podStartE2EDuration="1.715733534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:36.714919236 +0000 UTC m=+637.043330622" watchObservedRunningTime="2026-10-30 18:47:36.715733534 +0000 UTC m=+637.044144913" Oct 30 18:47:37.433789 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:37.433749 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:37.438668 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:37.438646 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:37.696820 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:37.696729 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:37.697667 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:37.697641 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:47:38.050338 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.050268 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t"] Oct 30 18:47:38.063684 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.058497 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:38.066291 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.066263 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t"] Oct 30 18:47:38.079541 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.079517 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx8b7\" (UniqueName: \"kubernetes.io/projected/29d33eb3-c35b-4c41-9723-925123b0cf98-kube-api-access-rx8b7\") pod \"payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t\" (UID: \"29d33eb3-c35b-4c41-9723-925123b0cf98\") " pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:38.079778 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.079672 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d33eb3-c35b-4c41-9723-925123b0cf98-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t\" (UID: \"29d33eb3-c35b-4c41-9723-925123b0cf98\") " pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:38.109250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.109228 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6156d5d6\"" Oct 30 18:47:38.126502 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.126468 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6156d5d6-dockercfg-2pkrx\"" Oct 30 18:47:38.180364 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.180304 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx8b7\" (UniqueName: \"kubernetes.io/projected/29d33eb3-c35b-4c41-9723-925123b0cf98-kube-api-access-rx8b7\") pod \"payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t\" (UID: \"29d33eb3-c35b-4c41-9723-925123b0cf98\") " pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:38.180600 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.180432 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d33eb3-c35b-4c41-9723-925123b0cf98-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t\" (UID: \"29d33eb3-c35b-4c41-9723-925123b0cf98\") " pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:38.181317 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.181296 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d33eb3-c35b-4c41-9723-925123b0cf98-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t\" (UID: \"29d33eb3-c35b-4c41-9723-925123b0cf98\") " pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:38.211639 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.211581 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx8b7\" (UniqueName: \"kubernetes.io/projected/29d33eb3-c35b-4c41-9723-925123b0cf98-kube-api-access-rx8b7\") pod \"payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t\" (UID: \"29d33eb3-c35b-4c41-9723-925123b0cf98\") " pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:38.379496 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.379403 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:38.659498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.659414 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t"] Oct 30 18:47:38.662999 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:47:38.662972 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d33eb3_c35b_4c41_9723_925123b0cf98.slice/crio-8d182ccbfbdd60ad28be1b924ab9350e914b7335ff9a90321afbca8e251a24db WatchSource:0}: Error finding container 8d182ccbfbdd60ad28be1b924ab9350e914b7335ff9a90321afbca8e251a24db: Status 404 returned error can't find the container with id 8d182ccbfbdd60ad28be1b924ab9350e914b7335ff9a90321afbca8e251a24db Oct 30 18:47:38.702423 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:38.702385 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" event={"ID":"29d33eb3-c35b-4c41-9723-925123b0cf98","Type":"ContainerStarted","Data":"8d182ccbfbdd60ad28be1b924ab9350e914b7335ff9a90321afbca8e251a24db"} Oct 30 18:47:40.142503 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.142469 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn"] Oct 30 18:47:40.151630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.151581 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:40.158756 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.158377 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn"] Oct 30 18:47:40.160644 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.160586 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ee87bf29-dockercfg-ggkl7\"" Oct 30 18:47:40.201723 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.201694 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpqj\" (UniqueName: \"kubernetes.io/projected/027bd3c6-4878-4d1e-9383-32af34dae07f-kube-api-access-srpqj\") pod \"payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn\" (UID: \"027bd3c6-4878-4d1e-9383-32af34dae07f\") " pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:40.201909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.201760 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027bd3c6-4878-4d1e-9383-32af34dae07f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn\" (UID: \"027bd3c6-4878-4d1e-9383-32af34dae07f\") " pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:40.223108 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.223073 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ee87bf29\"" Oct 30 18:47:40.302578 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.302503 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027bd3c6-4878-4d1e-9383-32af34dae07f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn\" (UID: \"027bd3c6-4878-4d1e-9383-32af34dae07f\") " pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:40.302755 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.302656 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpqj\" (UniqueName: \"kubernetes.io/projected/027bd3c6-4878-4d1e-9383-32af34dae07f-kube-api-access-srpqj\") pod \"payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn\" (UID: \"027bd3c6-4878-4d1e-9383-32af34dae07f\") " pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:40.303261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.303240 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027bd3c6-4878-4d1e-9383-32af34dae07f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn\" (UID: \"027bd3c6-4878-4d1e-9383-32af34dae07f\") " pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:40.311962 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.311936 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpqj\" (UniqueName: \"kubernetes.io/projected/027bd3c6-4878-4d1e-9383-32af34dae07f-kube-api-access-srpqj\") pod \"payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn\" (UID: \"027bd3c6-4878-4d1e-9383-32af34dae07f\") " pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:40.474327 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.474245 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:40.757949 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:40.757900 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn"] Oct 30 18:47:40.760466 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:47:40.760435 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027bd3c6_4878_4d1e_9383_32af34dae07f.slice/crio-da4668d9d9b4614b280c1bc7db6a1dbc1fc31d170877441e01c26a4c8ae1ced1 WatchSource:0}: Error finding container da4668d9d9b4614b280c1bc7db6a1dbc1fc31d170877441e01c26a4c8ae1ced1: Status 404 returned error can't find the container with id da4668d9d9b4614b280c1bc7db6a1dbc1fc31d170877441e01c26a4c8ae1ced1 Oct 30 18:47:41.340660 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.340623 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:47:41.720641 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.720584 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" event={"ID":"027bd3c6-4878-4d1e-9383-32af34dae07f","Type":"ContainerStarted","Data":"32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe"} Oct 30 18:47:41.720825 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.720659 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" event={"ID":"027bd3c6-4878-4d1e-9383-32af34dae07f","Type":"ContainerStarted","Data":"da4668d9d9b4614b280c1bc7db6a1dbc1fc31d170877441e01c26a4c8ae1ced1"} Oct 30 18:47:41.720825 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.720800 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:41.722100 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.722075 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" event={"ID":"29d33eb3-c35b-4c41-9723-925123b0cf98","Type":"ContainerStarted","Data":"c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772"} Oct 30 18:47:41.722208 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.722197 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:41.741573 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.741534 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" podStartSLOduration=1.332833301 podStartE2EDuration="1.741520937s" totalImagesPullingTime="408.687636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:41.73966061 +0000 UTC m=+642.068071986" watchObservedRunningTime="2026-10-30 18:47:41.741520937 +0000 UTC m=+642.069932316" Oct 30 18:47:41.769416 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.769368 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" podStartSLOduration=1.554034575 podStartE2EDuration="3.769356816s" totalImagesPullingTime="2.215322241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:41.76763165 +0000 UTC m=+642.096043035" watchObservedRunningTime="2026-10-30 18:47:41.769356816 +0000 UTC m=+642.097768201" Oct 30 18:47:41.965048 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.965007 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46"] Oct 30 18:47:41.969260 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.969237 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:41.980621 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.980582 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e4242163-serving-cert\"" Oct 30 18:47:41.980733 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.980724 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:47:41.985485 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.985463 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pd8db\"" Oct 30 18:47:41.987645 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.986031 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:47:41.987645 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.986120 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:47:41.987645 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:41.987491 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46"] Oct 30 18:47:42.019454 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.019423 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.019594 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.019473 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-openshift-service-ca\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.019594 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.019571 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wg5\" (UniqueName: \"kubernetes.io/projected/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-kube-api-access-w4wg5\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.019725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.019707 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-metrics-certs\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.120381 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.120347 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-openshift-service-ca\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.120584 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.120406 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wg5\" (UniqueName: \"kubernetes.io/projected/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-kube-api-access-w4wg5\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.120584 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.120469 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-metrics-certs\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.120584 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.120522 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.120749 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:47:42.120651 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-serving-cert: secret "maas-api-e2e-ait-e4242163-serving-cert" not found Oct 30 18:47:42.120749 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:47:42.120736 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls podName:fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01 nodeName:}" failed. No retries permitted until 2026-10-30 18:47:42.620714853 +0000 UTC m=+642.949126230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls") pod "maas-api-e2e-ait-e4242163-668f98b4f-kts46" (UID: "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01") : secret "maas-api-e2e-ait-e4242163-serving-cert" not found Oct 30 18:47:42.121082 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.121063 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-openshift-service-ca\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.123216 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.123195 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-metrics-certs\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.136998 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.136974 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wg5\" (UniqueName: \"kubernetes.io/projected/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-kube-api-access-w4wg5\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.624831 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.624797 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.627260 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.627232 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls\") pod \"maas-api-e2e-ait-e4242163-668f98b4f-kts46\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:42.883418 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:42.883319 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:43.176091 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:43.176060 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46"] Oct 30 18:47:43.179220 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:47:43.179190 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc39f1a5_74e3_41a5_931e_ce8e6cc8ec01.slice/crio-7720cc3987861c3f70f556dd139523b53d7b931aee57ba8c76d856c5b8bb5ce0 WatchSource:0}: Error finding container 7720cc3987861c3f70f556dd139523b53d7b931aee57ba8c76d856c5b8bb5ce0: Status 404 returned error can't find the container with id 7720cc3987861c3f70f556dd139523b53d7b931aee57ba8c76d856c5b8bb5ce0 Oct 30 18:47:43.733686 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:43.733597 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" event={"ID":"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01","Type":"ContainerStarted","Data":"7720cc3987861c3f70f556dd139523b53d7b931aee57ba8c76d856c5b8bb5ce0"} Oct 30 18:47:44.740107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:44.740074 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" event={"ID":"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01","Type":"ContainerStarted","Data":"3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c"} Oct 30 18:47:44.740504 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:44.740157 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:44.782263 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:44.782212 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" podStartSLOduration=3.29265832 podStartE2EDuration="3.782198745s" totalImagesPullingTime="489.540425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:44.779319955 +0000 UTC m=+645.107731339" watchObservedRunningTime="2026-10-30 18:47:44.782198745 +0000 UTC m=+645.110610127" Oct 30 18:47:47.705491 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.705460 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf"] Oct 30 18:47:47.709252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.709231 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:47.718285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.718256 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f951dfa9\"" Oct 30 18:47:47.720163 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.720142 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf"] Oct 30 18:47:47.776228 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.776187 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmzz\" (UniqueName: \"kubernetes.io/projected/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-kube-api-access-sxmzz\") pod \"payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf\" (UID: \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\") " pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:47.776407 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.776336 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-plugins-config-volume\") pod \"payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf\" (UID: \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\") " pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:47.877770 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.877735 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmzz\" (UniqueName: \"kubernetes.io/projected/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-kube-api-access-sxmzz\") pod \"payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf\" (UID: \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\") " pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:47.877987 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.877838 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-plugins-config-volume\") pod \"payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf\" (UID: \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\") " pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:47.878897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.878871 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-plugins-config-volume\") pod \"payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf\" (UID: \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\") " pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:47.886335 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:47.886311 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmzz\" (UniqueName: \"kubernetes.io/projected/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-kube-api-access-sxmzz\") pod \"payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf\" (UID: \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\") " pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:48.022872 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:48.022784 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:48.306832 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:47:48.306799 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c4b3e8_0571_4907_bbaa_56ffcdfc71c3.slice/crio-aad04ad58cec2c22a6bc78059211c38b24bf1000a83b319ab0000565377b1578 WatchSource:0}: Error finding container aad04ad58cec2c22a6bc78059211c38b24bf1000a83b319ab0000565377b1578: Status 404 returned error can't find the container with id aad04ad58cec2c22a6bc78059211c38b24bf1000a83b319ab0000565377b1578 Oct 30 18:47:48.312156 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:48.312131 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf"] Oct 30 18:47:48.764218 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:48.764184 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" event={"ID":"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3","Type":"ContainerStarted","Data":"5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b"} Oct 30 18:47:48.764722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:48.764232 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" event={"ID":"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3","Type":"ContainerStarted","Data":"aad04ad58cec2c22a6bc78059211c38b24bf1000a83b319ab0000565377b1578"} Oct 30 18:47:48.764722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:48.764286 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:48.781822 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:48.781777 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" podStartSLOduration=1.781762444 podStartE2EDuration="1.781762444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:48.78077044 +0000 UTC m=+649.109181826" watchObservedRunningTime="2026-10-30 18:47:48.781762444 +0000 UTC m=+649.110173828" Oct 30 18:47:49.971972 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:49.971934 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:47:50.751034 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:50.751002 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:47:51.361634 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.361587 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5"] Oct 30 18:47:51.365285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.365262 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.368931 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.368911 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f951dfa9-serving-cert\"" Oct 30 18:47:51.381320 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.381301 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5"] Oct 30 18:47:51.422655 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.422580 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9cb998-14d7-47da-8316-7be25c680fad-openshift-service-ca\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.422866 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.422758 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqzgd\" (UniqueName: \"kubernetes.io/projected/8a9cb998-14d7-47da-8316-7be25c680fad-kube-api-access-cqzgd\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.422866 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.422859 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-metrics-certs\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.423005 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.422938 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-maas-api-tls\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.523770 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.523720 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-maas-api-tls\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.523962 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.523823 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9cb998-14d7-47da-8316-7be25c680fad-openshift-service-ca\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.523962 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.523876 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqzgd\" (UniqueName: \"kubernetes.io/projected/8a9cb998-14d7-47da-8316-7be25c680fad-kube-api-access-cqzgd\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.523962 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.523950 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-metrics-certs\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.524679 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.524653 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9cb998-14d7-47da-8316-7be25c680fad-openshift-service-ca\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.526773 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.526752 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-maas-api-tls\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.526915 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.526897 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-metrics-certs\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.532967 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.532944 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqzgd\" (UniqueName: \"kubernetes.io/projected/8a9cb998-14d7-47da-8316-7be25c680fad-kube-api-access-cqzgd\") pod \"maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.680522 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.680491 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:51.985946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:51.985891 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5"] Oct 30 18:47:51.988678 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:47:51.988643 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9cb998_14d7_47da_8316_7be25c680fad.slice/crio-400e34b3d7a8d32ba2f435a0827a8c3a656d83ce8a911ce207923ce316528698 WatchSource:0}: Error finding container 400e34b3d7a8d32ba2f435a0827a8c3a656d83ce8a911ce207923ce316528698: Status 404 returned error can't find the container with id 400e34b3d7a8d32ba2f435a0827a8c3a656d83ce8a911ce207923ce316528698 Oct 30 18:47:52.729274 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:52.729198 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:47:52.729274 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:52.729265 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:47:52.792480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:52.792447 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" event={"ID":"8a9cb998-14d7-47da-8316-7be25c680fad","Type":"ContainerStarted","Data":"6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4"} Oct 30 18:47:52.792645 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:52.792505 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" event={"ID":"8a9cb998-14d7-47da-8316-7be25c680fad","Type":"ContainerStarted","Data":"400e34b3d7a8d32ba2f435a0827a8c3a656d83ce8a911ce207923ce316528698"} Oct 30 18:47:52.792645 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:52.792536 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:52.821245 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:52.821196 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" podStartSLOduration=1.363039068 podStartE2EDuration="1.821181371s" totalImagesPullingTime="458.142303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:52.820084052 +0000 UTC m=+653.148495440" watchObservedRunningTime="2026-10-30 18:47:52.821181371 +0000 UTC m=+653.149592750" Oct 30 18:47:57.779640 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:47:57.779572 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:47:58.829962 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:58.829925 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:47:58.847360 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:58.847331 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:47:59.772378 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:59.772301 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:47:59.854282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:59.854203 5959 generic.go:356] "Generic (PLEG): container finished" podID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerID="654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c" exitCode=0 Oct 30 18:47:59.854282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:59.854255 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" event={"ID":"ac6d9f29-13d9-4411-9b29-d3b4c56221e3","Type":"ContainerDied","Data":"654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c"} Oct 30 18:47:59.854706 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:59.854669 5959 scope.go:122] "RemoveContainer" containerID="654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c" Oct 30 18:47:59.856052 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:59.855999 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:59.856161 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:59.856122 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:47:59.856203 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:47:59.856171 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:48:00.576876 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.576842 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9"] Oct 30 18:48:00.580578 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.580558 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.584265 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.584244 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:48:00.587368 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.587220 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\"" Oct 30 18:48:00.590282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.590262 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9"] Oct 30 18:48:00.728352 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.728317 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.728528 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.728357 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.728528 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.728476 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcjq5\" (UniqueName: \"kubernetes.io/projected/e7f63711-15ee-4841-876e-a58ea0d1d9c6-kube-api-access-wcjq5\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.829139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.829048 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcjq5\" (UniqueName: \"kubernetes.io/projected/e7f63711-15ee-4841-876e-a58ea0d1d9c6-kube-api-access-wcjq5\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.829313 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.829150 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.829313 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.829171 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.829874 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.829853 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.831793 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.831772 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.837994 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.837970 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcjq5\" (UniqueName: \"kubernetes.io/projected/e7f63711-15ee-4841-876e-a58ea0d1d9c6-kube-api-access-wcjq5\") pod \"maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:00.860939 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.860911 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" event={"ID":"ac6d9f29-13d9-4411-9b29-d3b4c56221e3","Type":"ContainerStarted","Data":"9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7"} Oct 30 18:48:00.895146 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:00.895121 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:01.217633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:01.203261 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9"] Oct 30 18:48:01.225071 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:01.225040 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f63711_15ee_4841_876e_a58ea0d1d9c6.slice/crio-ad653cc3c64629d2f12de733f1134e2f62f1921517d4361f6a0f691bf49f43f6 WatchSource:0}: Error finding container ad653cc3c64629d2f12de733f1134e2f62f1921517d4361f6a0f691bf49f43f6: Status 404 returned error can't find the container with id ad653cc3c64629d2f12de733f1134e2f62f1921517d4361f6a0f691bf49f43f6 Oct 30 18:48:01.434054 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:01.434015 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:48:01.434457 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:01.434425 5959 patch_prober.go:29] interesting pod/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:48:01.434591 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:01.434488 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:48:01.867250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:01.867207 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" event={"ID":"e7f63711-15ee-4841-876e-a58ea0d1d9c6","Type":"ContainerStarted","Data":"ad653cc3c64629d2f12de733f1134e2f62f1921517d4361f6a0f691bf49f43f6"} Oct 30 18:48:02.434799 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:02.434764 5959 patch_prober.go:29] interesting pod/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:48:02.435000 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:02.434828 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:48:02.515632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:02.507353 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t"] Oct 30 18:48:03.433949 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:03.433919 5959 patch_prober.go:29] interesting pod/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:48:03.434330 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:03.433973 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:48:04.434170 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:04.434139 5959 patch_prober.go:29] interesting pod/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:48:04.434474 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:04.434181 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:48:04.887918 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:04.887877 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" event={"ID":"e7f63711-15ee-4841-876e-a58ea0d1d9c6","Type":"ContainerStarted","Data":"c3966c6486c90df307511f1dd629e2ded354cd13010367bbfde84513c14e3f1d"} Oct 30 18:48:04.907679 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:04.906772 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" podStartSLOduration=2.043043885 podStartE2EDuration="4.906754586s" totalImagesPullingTime="2.863710701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:04.90596016 +0000 UTC m=+665.234371573" watchObservedRunningTime="2026-10-30 18:48:04.906754586 +0000 UTC m=+665.235165972" Oct 30 18:48:05.433831 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:05.433795 5959 patch_prober.go:29] interesting pod/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:48:05.434008 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:05.433847 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:48:05.895885 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:05.895833 5959 generic.go:356] "Generic (PLEG): container finished" podID="e7f63711-15ee-4841-876e-a58ea0d1d9c6" containerID="c3966c6486c90df307511f1dd629e2ded354cd13010367bbfde84513c14e3f1d" exitCode=0 Oct 30 18:48:05.896377 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:05.895968 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" event={"ID":"e7f63711-15ee-4841-876e-a58ea0d1d9c6","Type":"ContainerDied","Data":"c3966c6486c90df307511f1dd629e2ded354cd13010367bbfde84513c14e3f1d"} Oct 30 18:48:06.438643 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:06.438593 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:48:06.438909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:06.438890 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:48:06.439457 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:06.439439 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:48:07.036566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.036537 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:07.116675 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.116646 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f63711-15ee-4841-876e-a58ea0d1d9c6-kube-api-access-wcjq5\" (UniqueName: \"kubernetes.io/projected/e7f63711-15ee-4841-876e-a58ea0d1d9c6-kube-api-access-wcjq5\") pod \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " Oct 30 18:48:07.116821 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.116717 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-openshift-ca\") pod \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " Oct 30 18:48:07.116821 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.116753 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-xks-ca\") pod \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\" (UID: \"e7f63711-15ee-4841-876e-a58ea0d1d9c6\") " Oct 30 18:48:07.117087 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.117053 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-openshift-ca" pod "e7f63711-15ee-4841-876e-a58ea0d1d9c6" (UID: "e7f63711-15ee-4841-876e-a58ea0d1d9c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:07.118853 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.118829 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f63711-15ee-4841-876e-a58ea0d1d9c6-kube-api-access-wcjq5" pod "e7f63711-15ee-4841-876e-a58ea0d1d9c6" (UID: "e7f63711-15ee-4841-876e-a58ea0d1d9c6"). InnerVolumeSpecName "kube-api-access-wcjq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:07.118934 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.118901 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-xks-ca" pod "e7f63711-15ee-4841-876e-a58ea0d1d9c6" (UID: "e7f63711-15ee-4841-876e-a58ea0d1d9c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:07.217695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.217634 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcjq5\" (UniqueName: \"kubernetes.io/projected/e7f63711-15ee-4841-876e-a58ea0d1d9c6-kube-api-access-wcjq5\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:07.217695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.217656 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-openshift-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:07.217695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.217666 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f63711-15ee-4841-876e-a58ea0d1d9c6-maas-api-xks-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:07.508612 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.508459 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" podUID="29d33eb3-c35b-4c41-9723-925123b0cf98" containerName="payload-processing" containerID="cri-o://c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772" gracePeriod=25 Oct 30 18:48:07.754733 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.754709 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:48:07.770986 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.770929 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:48:07.905941 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.905911 5959 generic.go:356] "Generic (PLEG): container finished" podID="29d33eb3-c35b-4c41-9723-925123b0cf98" containerID="c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772" exitCode=0 Oct 30 18:48:07.906110 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.905957 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" Oct 30 18:48:07.906110 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.905991 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" event={"ID":"29d33eb3-c35b-4c41-9723-925123b0cf98","Type":"ContainerDied","Data":"c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772"} Oct 30 18:48:07.906110 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.906047 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t" event={"ID":"29d33eb3-c35b-4c41-9723-925123b0cf98","Type":"ContainerDied","Data":"8d182ccbfbdd60ad28be1b924ab9350e914b7335ff9a90321afbca8e251a24db"} Oct 30 18:48:07.906110 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.906076 5959 scope.go:122] "RemoveContainer" containerID="c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772" Oct 30 18:48:07.907419 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.907401 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" Oct 30 18:48:07.907480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.907439 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9" event={"ID":"e7f63711-15ee-4841-876e-a58ea0d1d9c6","Type":"ContainerDied","Data":"ad653cc3c64629d2f12de733f1134e2f62f1921517d4361f6a0f691bf49f43f6"} Oct 30 18:48:07.907480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.907475 5959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad653cc3c64629d2f12de733f1134e2f62f1921517d4361f6a0f691bf49f43f6" Oct 30 18:48:07.915101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.915083 5959 scope.go:122] "RemoveContainer" containerID="c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772" Oct 30 18:48:07.915393 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:07.915374 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772\": container with ID starting with c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772 not found: ID does not exist" containerID="c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772" Oct 30 18:48:07.915453 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.915400 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772"} err="failed to get container status \"c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772\": rpc error: code = NotFound desc = could not find container \"c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772\": container with ID starting with c5cc4dead8b7cf275bf933e7aa8f41278b74012bc3c0058f17c8a7d5783ca772 not found: ID does not exist" Oct 30 18:48:07.925806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.925783 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d33eb3-c35b-4c41-9723-925123b0cf98-kube-api-access-rx8b7\" (UniqueName: \"kubernetes.io/projected/29d33eb3-c35b-4c41-9723-925123b0cf98-kube-api-access-rx8b7\") pod \"29d33eb3-c35b-4c41-9723-925123b0cf98\" (UID: \"29d33eb3-c35b-4c41-9723-925123b0cf98\") " Oct 30 18:48:07.925900 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.925810 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d33eb3-c35b-4c41-9723-925123b0cf98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d33eb3-c35b-4c41-9723-925123b0cf98-plugins-config-volume\") pod \"29d33eb3-c35b-4c41-9723-925123b0cf98\" (UID: \"29d33eb3-c35b-4c41-9723-925123b0cf98\") " Oct 30 18:48:07.926234 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.926195 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d33eb3-c35b-4c41-9723-925123b0cf98-plugins-config-volume" pod "29d33eb3-c35b-4c41-9723-925123b0cf98" (UID: "29d33eb3-c35b-4c41-9723-925123b0cf98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:07.928017 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:07.927995 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d33eb3-c35b-4c41-9723-925123b0cf98-kube-api-access-rx8b7" pod "29d33eb3-c35b-4c41-9723-925123b0cf98" (UID: "29d33eb3-c35b-4c41-9723-925123b0cf98"). InnerVolumeSpecName "kube-api-access-rx8b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:08.026988 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:08.026933 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx8b7\" (UniqueName: \"kubernetes.io/projected/29d33eb3-c35b-4c41-9723-925123b0cf98-kube-api-access-rx8b7\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:08.026988 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:08.026955 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d33eb3-c35b-4c41-9723-925123b0cf98-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:08.233091 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:08.233052 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t"] Oct 30 18:48:08.236924 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:08.236903 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6156d5d6-fb57cdc98-7bh9t"] Oct 30 18:48:09.472756 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:09.472717 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn"] Oct 30 18:48:09.796703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:09.796596 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d33eb3-c35b-4c41-9723-925123b0cf98" path="/var/lib/kubelet/pods/29d33eb3-c35b-4c41-9723-925123b0cf98/volumes" Oct 30 18:48:10.294372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.294337 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46"] Oct 30 18:48:10.294561 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.294532 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" podUID="fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" containerName="maas-api" containerID="cri-o://3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c" gracePeriod=30 Oct 30 18:48:10.572264 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.572234 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:48:10.752563 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.752525 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-kube-api-access-w4wg5\" (UniqueName: \"kubernetes.io/projected/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-kube-api-access-w4wg5\") pod \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " Oct 30 18:48:10.752769 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.752635 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-openshift-service-ca\") pod \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " Oct 30 18:48:10.752769 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.752667 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-metrics-certs\") pod \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " Oct 30 18:48:10.752861 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.752814 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls\") pod \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\" (UID: \"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01\") " Oct 30 18:48:10.753217 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.753190 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-openshift-service-ca" pod "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" (UID: "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:10.755133 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.755103 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-metrics-certs" pod "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" (UID: "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:10.755231 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.755140 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls" pod "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" (UID: "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:10.755275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.755263 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-kube-api-access-w4wg5" pod "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" (UID: "fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01"). InnerVolumeSpecName "kube-api-access-w4wg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:10.853850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.853773 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:10.853850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.853800 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4wg5\" (UniqueName: \"kubernetes.io/projected/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-kube-api-access-w4wg5\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:10.853850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.853812 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:10.853850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.853823 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:10.924881 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.924847 5959 generic.go:356] "Generic (PLEG): container finished" podID="fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" containerID="3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c" exitCode=0 Oct 30 18:48:10.925044 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.924892 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" Oct 30 18:48:10.925044 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.924908 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" event={"ID":"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01","Type":"ContainerDied","Data":"3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c"} Oct 30 18:48:10.925044 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.924961 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46" event={"ID":"fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01","Type":"ContainerDied","Data":"7720cc3987861c3f70f556dd139523b53d7b931aee57ba8c76d856c5b8bb5ce0"} Oct 30 18:48:10.925044 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.924985 5959 scope.go:122] "RemoveContainer" containerID="3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c" Oct 30 18:48:10.934674 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.934660 5959 scope.go:122] "RemoveContainer" containerID="3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c" Oct 30 18:48:10.934965 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:10.934947 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c\": container with ID starting with 3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c not found: ID does not exist" containerID="3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c" Oct 30 18:48:10.935009 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.934974 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c"} err="failed to get container status \"3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c\": rpc error: code = NotFound desc = could not find container \"3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c\": container with ID starting with 3482ff28cdb5a33457f07b4c606811c3273298b971e9091056b087b4db85649c not found: ID does not exist" Oct 30 18:48:10.945964 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.945944 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46"] Oct 30 18:48:10.951640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:10.951621 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4242163-668f98b4f-kts46"] Oct 30 18:48:11.251118 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:11.251080 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf"] Oct 30 18:48:11.724620 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:11.724574 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k"] Oct 30 18:48:11.725011 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:11.724781 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" containerID="cri-o://9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7" gracePeriod=30 Oct 30 18:48:11.795652 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:11.795598 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" path="/var/lib/kubelet/pods/fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01/volumes" Oct 30 18:48:11.875974 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:11.875938 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5"] Oct 30 18:48:11.876190 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:11.876139 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" podUID="8a9cb998-14d7-47da-8316-7be25c680fad" containerName="maas-api" containerID="cri-o://6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4" gracePeriod=30 Oct 30 18:48:12.127312 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.127288 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:48:12.222137 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.222100 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9"] Oct 30 18:48:12.223973 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.223942 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f951dfa9-hkfp9"] Oct 30 18:48:12.269535 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.269470 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9cb998-14d7-47da-8316-7be25c680fad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9cb998-14d7-47da-8316-7be25c680fad-openshift-service-ca\") pod \"8a9cb998-14d7-47da-8316-7be25c680fad\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " Oct 30 18:48:12.269535 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.269499 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9cb998-14d7-47da-8316-7be25c680fad-kube-api-access-cqzgd\" (UniqueName: \"kubernetes.io/projected/8a9cb998-14d7-47da-8316-7be25c680fad-kube-api-access-cqzgd\") pod \"8a9cb998-14d7-47da-8316-7be25c680fad\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " Oct 30 18:48:12.269693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.269559 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-maas-api-tls\") pod \"8a9cb998-14d7-47da-8316-7be25c680fad\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " Oct 30 18:48:12.269693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.269648 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-metrics-certs\") pod \"8a9cb998-14d7-47da-8316-7be25c680fad\" (UID: \"8a9cb998-14d7-47da-8316-7be25c680fad\") " Oct 30 18:48:12.269900 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.269874 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9cb998-14d7-47da-8316-7be25c680fad-openshift-service-ca" pod "8a9cb998-14d7-47da-8316-7be25c680fad" (UID: "8a9cb998-14d7-47da-8316-7be25c680fad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:12.271816 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.271785 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-metrics-certs" pod "8a9cb998-14d7-47da-8316-7be25c680fad" (UID: "8a9cb998-14d7-47da-8316-7be25c680fad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:12.271911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.271843 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-maas-api-tls" pod "8a9cb998-14d7-47da-8316-7be25c680fad" (UID: "8a9cb998-14d7-47da-8316-7be25c680fad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:12.271911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.271861 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9cb998-14d7-47da-8316-7be25c680fad-kube-api-access-cqzgd" pod "8a9cb998-14d7-47da-8316-7be25c680fad" (UID: "8a9cb998-14d7-47da-8316-7be25c680fad"). InnerVolumeSpecName "kube-api-access-cqzgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:12.370520 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.370469 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a9cb998-14d7-47da-8316-7be25c680fad-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:12.370520 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.370515 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqzgd\" (UniqueName: \"kubernetes.io/projected/8a9cb998-14d7-47da-8316-7be25c680fad-kube-api-access-cqzgd\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:12.370785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.370532 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:12.370785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.370546 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9cb998-14d7-47da-8316-7be25c680fad-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:12.938486 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.938447 5959 generic.go:356] "Generic (PLEG): container finished" podID="8a9cb998-14d7-47da-8316-7be25c680fad" containerID="6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4" exitCode=0 Oct 30 18:48:12.938989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.938549 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" event={"ID":"8a9cb998-14d7-47da-8316-7be25c680fad","Type":"ContainerDied","Data":"6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4"} Oct 30 18:48:12.938989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.938550 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" Oct 30 18:48:12.938989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.938598 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5" event={"ID":"8a9cb998-14d7-47da-8316-7be25c680fad","Type":"ContainerDied","Data":"400e34b3d7a8d32ba2f435a0827a8c3a656d83ce8a911ce207923ce316528698"} Oct 30 18:48:12.938989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.938657 5959 scope.go:122] "RemoveContainer" containerID="6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4" Oct 30 18:48:12.956460 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.956438 5959 scope.go:122] "RemoveContainer" containerID="6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4" Oct 30 18:48:12.956765 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:12.956745 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4\": container with ID starting with 6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4 not found: ID does not exist" containerID="6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4" Oct 30 18:48:12.956854 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.956772 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4"} err="failed to get container status \"6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4\": rpc error: code = NotFound desc = could not find container \"6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4\": container with ID starting with 6883dce716ec7e7d3e614f9ee9e20427320babf1c7424b810e643671250154e4 not found: ID does not exist" Oct 30 18:48:12.965693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.965670 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5"] Oct 30 18:48:12.973933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:12.973910 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f951dfa9-57c64f77bd-flff5"] Oct 30 18:48:13.148894 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.148863 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj"] Oct 30 18:48:13.149449 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149421 5959 cpu_manager.go:414] "containerMap: removing container" podUID="29d33eb3-c35b-4c41-9723-925123b0cf98" containerName="payload-processing" Oct 30 18:48:13.149449 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149449 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d33eb3-c35b-4c41-9723-925123b0cf98" containerName="payload-processing" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149474 5959 cpu_manager.go:414] "containerMap: removing container" podUID="e7f63711-15ee-4841-876e-a58ea0d1d9c6" containerName="revoke-keys" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149480 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f63711-15ee-4841-876e-a58ea0d1d9c6" containerName="revoke-keys" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149490 5959 cpu_manager.go:414] "containerMap: removing container" podUID="fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" containerName="maas-api" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149495 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" containerName="maas-api" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149501 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8a9cb998-14d7-47da-8316-7be25c680fad" containerName="maas-api" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149508 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9cb998-14d7-47da-8316-7be25c680fad" containerName="maas-api" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149591 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d33eb3-c35b-4c41-9723-925123b0cf98" containerName="payload-processing" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149599 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f63711-15ee-4841-876e-a58ea0d1d9c6" containerName="revoke-keys" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149627 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc39f1a5-74e3-41a5-931e-ce8e6cc8ec01" containerName="maas-api" Oct 30 18:48:13.149647 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.149633 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9cb998-14d7-47da-8316-7be25c680fad" containerName="maas-api" Oct 30 18:48:13.154335 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.154310 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.163817 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.163793 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-2889dd-serving-cert\"" Oct 30 18:48:13.170693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.167718 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj"] Oct 30 18:48:13.170693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.170569 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:48:13.170693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.170667 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:48:13.201959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.201871 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:48:13.201959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.201891 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pd8db\"" Oct 30 18:48:13.280177 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.280117 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrqq\" (UniqueName: \"kubernetes.io/projected/424643ce-2ec3-462d-82a8-f5a61ea7a329-kube-api-access-vbrqq\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.280388 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.280360 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424643ce-2ec3-462d-82a8-f5a61ea7a329-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.280473 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.280428 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.280527 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.280476 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-metrics-certs\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.381705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.381672 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424643ce-2ec3-462d-82a8-f5a61ea7a329-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.381884 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.381745 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.381884 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.381774 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-metrics-certs\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.381884 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.381833 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrqq\" (UniqueName: \"kubernetes.io/projected/424643ce-2ec3-462d-82a8-f5a61ea7a329-kube-api-access-vbrqq\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.382393 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.382369 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424643ce-2ec3-462d-82a8-f5a61ea7a329-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.384439 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.384418 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-metrics-certs\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.384536 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.384459 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.390156 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.390133 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrqq\" (UniqueName: \"kubernetes.io/projected/424643ce-2ec3-462d-82a8-f5a61ea7a329-kube-api-access-vbrqq\") pod \"maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.473900 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.473815 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:13.780501 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.780423 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj"] Oct 30 18:48:13.785328 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:13.785300 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424643ce_2ec3_462d_82a8_f5a61ea7a329.slice/crio-a58969f7f7ea43c61ab4351a5a1cbbdf9c34926ea84e80480503c6e619db05e4 WatchSource:0}: Error finding container a58969f7f7ea43c61ab4351a5a1cbbdf9c34926ea84e80480503c6e619db05e4: Status 404 returned error can't find the container with id a58969f7f7ea43c61ab4351a5a1cbbdf9c34926ea84e80480503c6e619db05e4 Oct 30 18:48:13.797680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.797650 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9cb998-14d7-47da-8316-7be25c680fad" path="/var/lib/kubelet/pods/8a9cb998-14d7-47da-8316-7be25c680fad/volumes" Oct 30 18:48:13.798032 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.798017 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f63711-15ee-4841-876e-a58ea0d1d9c6" path="/var/lib/kubelet/pods/e7f63711-15ee-4841-876e-a58ea0d1d9c6/volumes" Oct 30 18:48:13.945898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:13.945857 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" event={"ID":"424643ce-2ec3-462d-82a8-f5a61ea7a329","Type":"ContainerStarted","Data":"a58969f7f7ea43c61ab4351a5a1cbbdf9c34926ea84e80480503c6e619db05e4"} Oct 30 18:48:14.473079 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.472990 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" podUID="027bd3c6-4878-4d1e-9383-32af34dae07f" containerName="payload-processing" containerID="cri-o://32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe" gracePeriod=25 Oct 30 18:48:14.726065 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.726004 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:48:14.898403 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.898369 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027bd3c6-4878-4d1e-9383-32af34dae07f-kube-api-access-srpqj\" (UniqueName: \"kubernetes.io/projected/027bd3c6-4878-4d1e-9383-32af34dae07f-kube-api-access-srpqj\") pod \"027bd3c6-4878-4d1e-9383-32af34dae07f\" (UID: \"027bd3c6-4878-4d1e-9383-32af34dae07f\") " Oct 30 18:48:14.898592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.898450 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027bd3c6-4878-4d1e-9383-32af34dae07f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027bd3c6-4878-4d1e-9383-32af34dae07f-plugins-config-volume\") pod \"027bd3c6-4878-4d1e-9383-32af34dae07f\" (UID: \"027bd3c6-4878-4d1e-9383-32af34dae07f\") " Oct 30 18:48:14.898839 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.898814 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027bd3c6-4878-4d1e-9383-32af34dae07f-plugins-config-volume" pod "027bd3c6-4878-4d1e-9383-32af34dae07f" (UID: "027bd3c6-4878-4d1e-9383-32af34dae07f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:14.900717 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.900692 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027bd3c6-4878-4d1e-9383-32af34dae07f-kube-api-access-srpqj" pod "027bd3c6-4878-4d1e-9383-32af34dae07f" (UID: "027bd3c6-4878-4d1e-9383-32af34dae07f"). InnerVolumeSpecName "kube-api-access-srpqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:14.952568 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.952539 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" event={"ID":"424643ce-2ec3-462d-82a8-f5a61ea7a329","Type":"ContainerStarted","Data":"dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612"} Oct 30 18:48:14.952971 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.952647 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:14.953948 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.953928 5959 generic.go:356] "Generic (PLEG): container finished" podID="027bd3c6-4878-4d1e-9383-32af34dae07f" containerID="32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe" exitCode=0 Oct 30 18:48:14.954055 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.953965 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" Oct 30 18:48:14.954055 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.953992 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" event={"ID":"027bd3c6-4878-4d1e-9383-32af34dae07f","Type":"ContainerDied","Data":"32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe"} Oct 30 18:48:14.954055 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.954023 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn" event={"ID":"027bd3c6-4878-4d1e-9383-32af34dae07f","Type":"ContainerDied","Data":"da4668d9d9b4614b280c1bc7db6a1dbc1fc31d170877441e01c26a4c8ae1ced1"} Oct 30 18:48:14.954055 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.954046 5959 scope.go:122] "RemoveContainer" containerID="32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe" Oct 30 18:48:14.964218 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.964204 5959 scope.go:122] "RemoveContainer" containerID="32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe" Oct 30 18:48:14.964477 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:14.964462 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe\": container with ID starting with 32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe not found: ID does not exist" containerID="32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe" Oct 30 18:48:14.964533 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.964485 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe"} err="failed to get container status \"32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe\": rpc error: code = NotFound desc = could not find container \"32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe\": container with ID starting with 32c309af9d27e6bbf50b4f5e87a1d34c46f57bc99ca7a2431017b326f39b37fe not found: ID does not exist" Oct 30 18:48:14.974478 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.974441 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" podStartSLOduration=1.576256112 podStartE2EDuration="1.974428331s" totalImagesPullingTime="398.172219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:14.973888988 +0000 UTC m=+675.302300373" watchObservedRunningTime="2026-10-30 18:48:14.974428331 +0000 UTC m=+675.302839708" Oct 30 18:48:14.989984 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.989935 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn"] Oct 30 18:48:14.993751 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.993731 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee87bf29-6fbff56fcd-jdszn"] Oct 30 18:48:14.999832 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.999807 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srpqj\" (UniqueName: \"kubernetes.io/projected/027bd3c6-4878-4d1e-9383-32af34dae07f-kube-api-access-srpqj\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:14.999832 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:14.999827 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027bd3c6-4878-4d1e-9383-32af34dae07f-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:15.776354 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:15.776316 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:48:15.795230 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:15.795192 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027bd3c6-4878-4d1e-9383-32af34dae07f" path="/var/lib/kubelet/pods/027bd3c6-4878-4d1e-9383-32af34dae07f/volumes" Oct 30 18:48:16.252328 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.252288 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" podUID="66c4b3e8-0571-4907-bbaa-56ffcdfc71c3" containerName="payload-processing" containerID="cri-o://5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b" gracePeriod=25 Oct 30 18:48:16.507848 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.507769 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:48:16.515487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.515464 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-plugins-config-volume\") pod \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\" (UID: \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\") " Oct 30 18:48:16.515624 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.515583 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-kube-api-access-sxmzz\" (UniqueName: \"kubernetes.io/projected/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-kube-api-access-sxmzz\") pod \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\" (UID: \"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3\") " Oct 30 18:48:16.515835 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.515802 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-plugins-config-volume" pod "66c4b3e8-0571-4907-bbaa-56ffcdfc71c3" (UID: "66c4b3e8-0571-4907-bbaa-56ffcdfc71c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:16.517735 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.517712 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-kube-api-access-sxmzz" pod "66c4b3e8-0571-4907-bbaa-56ffcdfc71c3" (UID: "66c4b3e8-0571-4907-bbaa-56ffcdfc71c3"). InnerVolumeSpecName "kube-api-access-sxmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:16.616707 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.616681 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxmzz\" (UniqueName: \"kubernetes.io/projected/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-kube-api-access-sxmzz\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:16.616707 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.616704 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:16.960570 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.960538 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv"] Oct 30 18:48:16.960987 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.960972 5959 cpu_manager.go:414] "containerMap: removing container" podUID="027bd3c6-4878-4d1e-9383-32af34dae07f" containerName="payload-processing" Oct 30 18:48:16.961039 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.960989 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027bd3c6-4878-4d1e-9383-32af34dae07f" containerName="payload-processing" Oct 30 18:48:16.961039 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.961000 5959 cpu_manager.go:414] "containerMap: removing container" podUID="66c4b3e8-0571-4907-bbaa-56ffcdfc71c3" containerName="payload-processing" Oct 30 18:48:16.961039 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.961005 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c4b3e8-0571-4907-bbaa-56ffcdfc71c3" containerName="payload-processing" Oct 30 18:48:16.961132 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.961114 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="027bd3c6-4878-4d1e-9383-32af34dae07f" containerName="payload-processing" Oct 30 18:48:16.961132 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.961123 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c4b3e8-0571-4907-bbaa-56ffcdfc71c3" containerName="payload-processing" Oct 30 18:48:16.963061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.963036 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:48:16.963991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.963974 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:16.976395 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.976374 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-39bde3-dockercfg-hq85c\"" Oct 30 18:48:16.976395 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.976378 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-39bde3\"" Oct 30 18:48:16.977537 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.977517 5959 generic.go:356] "Generic (PLEG): container finished" podID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerID="9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7" exitCode=0 Oct 30 18:48:16.977685 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.977563 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" Oct 30 18:48:16.977685 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.977617 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" event={"ID":"ac6d9f29-13d9-4411-9b29-d3b4c56221e3","Type":"ContainerDied","Data":"9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7"} Oct 30 18:48:16.977685 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.977669 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k" event={"ID":"ac6d9f29-13d9-4411-9b29-d3b4c56221e3","Type":"ContainerDied","Data":"fdb44facc68dd09fe5374aaf33984c8e0998b8c9735160dc662d7f12b6ae44be"} Oct 30 18:48:16.977817 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.977700 5959 scope.go:122] "RemoveContainer" containerID="9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7" Oct 30 18:48:16.979120 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.979075 5959 generic.go:356] "Generic (PLEG): container finished" podID="66c4b3e8-0571-4907-bbaa-56ffcdfc71c3" containerID="5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b" exitCode=0 Oct 30 18:48:16.979209 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.979123 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" Oct 30 18:48:16.979209 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.979153 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" event={"ID":"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3","Type":"ContainerDied","Data":"5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b"} Oct 30 18:48:16.979209 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.979196 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf" event={"ID":"66c4b3e8-0571-4907-bbaa-56ffcdfc71c3","Type":"ContainerDied","Data":"aad04ad58cec2c22a6bc78059211c38b24bf1000a83b319ab0000565377b1578"} Oct 30 18:48:16.986827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.986805 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv"] Oct 30 18:48:16.987537 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.987519 5959 scope.go:122] "RemoveContainer" containerID="654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c" Oct 30 18:48:16.997572 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.997548 5959 scope.go:122] "RemoveContainer" containerID="9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7" Oct 30 18:48:16.997926 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:16.997910 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7\": container with ID starting with 9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7 not found: ID does not exist" containerID="9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7" Oct 30 18:48:16.998008 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.997938 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7"} err="failed to get container status \"9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7\": rpc error: code = NotFound desc = could not find container \"9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7\": container with ID starting with 9cec591bd8acc704fcb0bad0f62a7de3508f42858ce247596cbcfc710af1cee7 not found: ID does not exist" Oct 30 18:48:16.998008 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.997962 5959 scope.go:122] "RemoveContainer" containerID="654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c" Oct 30 18:48:16.998269 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:16.998250 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c\": container with ID starting with 654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c not found: ID does not exist" containerID="654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c" Oct 30 18:48:16.998307 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.998275 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c"} err="failed to get container status \"654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c\": rpc error: code = NotFound desc = could not find container \"654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c\": container with ID starting with 654ea9bfe7ec46d23eb8d0a0d5cebec8d105bb3a86236d2457224657a1aa704c not found: ID does not exist" Oct 30 18:48:16.998307 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:16.998290 5959 scope.go:122] "RemoveContainer" containerID="5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b" Oct 30 18:48:17.007085 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.007065 5959 scope.go:122] "RemoveContainer" containerID="5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b" Oct 30 18:48:17.007360 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:17.007338 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b\": container with ID starting with 5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b not found: ID does not exist" containerID="5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b" Oct 30 18:48:17.007421 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.007371 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b"} err="failed to get container status \"5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b\": rpc error: code = NotFound desc = could not find container \"5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b\": container with ID starting with 5dfd869bed217e25dc877e50f74e778bcd1b1f8a1101f99c0ca1143ab3e11d7b not found: ID does not exist" Oct 30 18:48:17.020860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.020840 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-data\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.020967 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.020886 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-certs\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.020967 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.020907 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-envoy\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.021085 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.021035 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-socket\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.021085 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.021074 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istiod-ca-cert\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.021664 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.021102 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-credential-socket\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.021664 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.021144 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-podinfo\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.021664 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.021181 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-token\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-token\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.021664 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.021217 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-certs" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:17.021664 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.021241 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-kube-api-access-56bh4\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-kube-api-access-56bh4\") pod \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\" (UID: \"ac6d9f29-13d9-4411-9b29-d3b4c56221e3\") " Oct 30 18:48:17.021914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.021815 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzrr\" (UniqueName: \"kubernetes.io/projected/108e4e52-68b1-458e-aca1-8706621e3a57-kube-api-access-klzrr\") pod \"payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv\" (UID: \"108e4e52-68b1-458e-aca1-8706621e3a57\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:17.022134 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.022107 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108e4e52-68b1-458e-aca1-8706621e3a57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv\" (UID: \"108e4e52-68b1-458e-aca1-8706621e3a57\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:17.022258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.022204 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.022320 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.022305 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-socket" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:17.022404 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.022343 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-data" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:17.022757 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.022492 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-credential-socket" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:17.022757 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.022695 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istiod-ca-cert" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:17.024560 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.024510 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-envoy" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:17.025095 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.025072 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-podinfo" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:48:17.025560 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.025540 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-kube-api-access-56bh4" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "kube-api-access-56bh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:17.025741 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.025723 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-token" pod "ac6d9f29-13d9-4411-9b29-d3b4c56221e3" (UID: "ac6d9f29-13d9-4411-9b29-d3b4c56221e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:17.051941 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.051920 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf"] Oct 30 18:48:17.066136 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.066116 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f951dfa9-57f46fd8db-pvmwf"] Oct 30 18:48:17.122901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.122846 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzrr\" (UniqueName: \"kubernetes.io/projected/108e4e52-68b1-458e-aca1-8706621e3a57-kube-api-access-klzrr\") pod \"payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv\" (UID: \"108e4e52-68b1-458e-aca1-8706621e3a57\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:17.123010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.122944 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108e4e52-68b1-458e-aca1-8706621e3a57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv\" (UID: \"108e4e52-68b1-458e-aca1-8706621e3a57\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:17.123078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123009 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.123078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123025 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.123078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123039 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.123078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123052 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.123078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123065 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.123078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123078 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bh4\" (UniqueName: \"kubernetes.io/projected/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-kube-api-access-56bh4\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.123310 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123092 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.123310 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123105 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6d9f29-13d9-4411-9b29-d3b4c56221e3-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:48:17.123774 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.123755 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108e4e52-68b1-458e-aca1-8706621e3a57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv\" (UID: \"108e4e52-68b1-458e-aca1-8706621e3a57\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:17.135959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.135940 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzrr\" (UniqueName: \"kubernetes.io/projected/108e4e52-68b1-458e-aca1-8706621e3a57-kube-api-access-klzrr\") pod \"payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv\" (UID: \"108e4e52-68b1-458e-aca1-8706621e3a57\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:17.278221 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.278191 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:17.303938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.303907 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k"] Oct 30 18:48:17.309075 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.309052 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ee87bf29-openshift-default-7d64c658c7-bqm5k"] Oct 30 18:48:17.553866 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.553837 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv"] Oct 30 18:48:17.557507 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:17.557475 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108e4e52_68b1_458e_aca1_8706621e3a57.slice/crio-1becb86c40af6831aed6bf194e11d9de27d8c8a6809cac2cef66e56ea958a29c WatchSource:0}: Error finding container 1becb86c40af6831aed6bf194e11d9de27d8c8a6809cac2cef66e56ea958a29c: Status 404 returned error can't find the container with id 1becb86c40af6831aed6bf194e11d9de27d8c8a6809cac2cef66e56ea958a29c Oct 30 18:48:17.711943 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.711913 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7"] Oct 30 18:48:17.712339 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.712326 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" Oct 30 18:48:17.712384 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.712341 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" Oct 30 18:48:17.712464 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.712454 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" Oct 30 18:48:17.715635 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.715599 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:17.721331 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.721311 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9de872\"" Oct 30 18:48:17.721452 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.721372 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9de872-dockercfg-v2g5s\"" Oct 30 18:48:17.728848 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.728819 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7"] Oct 30 18:48:17.729170 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.729143 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bd45ad-4844-4771-ab04-be34f44f15f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7\" (UID: \"23bd45ad-4844-4771-ab04-be34f44f15f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:17.729284 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.729267 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xmh\" (UniqueName: \"kubernetes.io/projected/23bd45ad-4844-4771-ab04-be34f44f15f2-kube-api-access-w2xmh\") pod \"payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7\" (UID: \"23bd45ad-4844-4771-ab04-be34f44f15f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:17.761701 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.761669 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp"] Oct 30 18:48:17.762126 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.762109 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" Oct 30 18:48:17.762126 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.762126 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" Oct 30 18:48:17.762279 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.762200 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" containerName="istio-proxy" Oct 30 18:48:17.765543 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.765518 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:17.778506 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.778470 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp"] Oct 30 18:48:17.795589 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.795564 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c4b3e8-0571-4907-bbaa-56ffcdfc71c3" path="/var/lib/kubelet/pods/66c4b3e8-0571-4907-bbaa-56ffcdfc71c3/volumes" Oct 30 18:48:17.796006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.795992 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6d9f29-13d9-4411-9b29-d3b4c56221e3" path="/var/lib/kubelet/pods/ac6d9f29-13d9-4411-9b29-d3b4c56221e3/volumes" Oct 30 18:48:17.830342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.830270 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md9c9\" (UniqueName: \"kubernetes.io/projected/cff7c99c-0d72-4542-a2ef-28910e83d0bf-kube-api-access-md9c9\") pod \"payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp\" (UID: \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:17.830342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.830311 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff7c99c-0d72-4542-a2ef-28910e83d0bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp\" (UID: \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:17.830512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.830397 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bd45ad-4844-4771-ab04-be34f44f15f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7\" (UID: \"23bd45ad-4844-4771-ab04-be34f44f15f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:17.830512 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.830470 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xmh\" (UniqueName: \"kubernetes.io/projected/23bd45ad-4844-4771-ab04-be34f44f15f2-kube-api-access-w2xmh\") pod \"payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7\" (UID: \"23bd45ad-4844-4771-ab04-be34f44f15f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:17.831030 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.831014 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bd45ad-4844-4771-ab04-be34f44f15f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7\" (UID: \"23bd45ad-4844-4771-ab04-be34f44f15f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:17.839505 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.839480 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xmh\" (UniqueName: \"kubernetes.io/projected/23bd45ad-4844-4771-ab04-be34f44f15f2-kube-api-access-w2xmh\") pod \"payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7\" (UID: \"23bd45ad-4844-4771-ab04-be34f44f15f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:17.931754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.931712 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md9c9\" (UniqueName: \"kubernetes.io/projected/cff7c99c-0d72-4542-a2ef-28910e83d0bf-kube-api-access-md9c9\") pod \"payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp\" (UID: \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:17.931883 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.931776 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff7c99c-0d72-4542-a2ef-28910e83d0bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp\" (UID: \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:17.932469 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.932449 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff7c99c-0d72-4542-a2ef-28910e83d0bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp\" (UID: \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:17.940344 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.940319 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md9c9\" (UniqueName: \"kubernetes.io/projected/cff7c99c-0d72-4542-a2ef-28910e83d0bf-kube-api-access-md9c9\") pod \"payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp\" (UID: \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:17.987758 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.987732 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" event={"ID":"108e4e52-68b1-458e-aca1-8706621e3a57","Type":"ContainerStarted","Data":"276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6"} Oct 30 18:48:17.987854 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.987775 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" event={"ID":"108e4e52-68b1-458e-aca1-8706621e3a57","Type":"ContainerStarted","Data":"1becb86c40af6831aed6bf194e11d9de27d8c8a6809cac2cef66e56ea958a29c"} Oct 30 18:48:17.987906 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:17.987876 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:18.009119 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.009080 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" podStartSLOduration=2.009066737 podStartE2EDuration="2.009066737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:18.008053444 +0000 UTC m=+678.336464855" watchObservedRunningTime="2026-10-30 18:48:18.009066737 +0000 UTC m=+678.337478120" Oct 30 18:48:18.031447 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.031417 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:18.083853 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.083832 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:18.311583 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.311554 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7"] Oct 30 18:48:18.316423 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:18.316386 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23bd45ad_4844_4771_ab04_be34f44f15f2.slice/crio-3ac6ed55df7e9172fcc860bbdb8c60c7d50c60f343c7762d23e1ef5b092b53bc WatchSource:0}: Error finding container 3ac6ed55df7e9172fcc860bbdb8c60c7d50c60f343c7762d23e1ef5b092b53bc: Status 404 returned error can't find the container with id 3ac6ed55df7e9172fcc860bbdb8c60c7d50c60f343c7762d23e1ef5b092b53bc Oct 30 18:48:18.378645 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.377650 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp"] Oct 30 18:48:18.383150 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:18.383125 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff7c99c_0d72_4542_a2ef_28910e83d0bf.slice/crio-d6446c47bab72b0fc6de393f8c0d634768fea3dd06c09855d4308a276ab2bf0a WatchSource:0}: Error finding container d6446c47bab72b0fc6de393f8c0d634768fea3dd06c09855d4308a276ab2bf0a: Status 404 returned error can't find the container with id d6446c47bab72b0fc6de393f8c0d634768fea3dd06c09855d4308a276ab2bf0a Oct 30 18:48:18.994978 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.994935 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" event={"ID":"cff7c99c-0d72-4542-a2ef-28910e83d0bf","Type":"ContainerStarted","Data":"d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead"} Oct 30 18:48:18.995219 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.995040 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" event={"ID":"cff7c99c-0d72-4542-a2ef-28910e83d0bf","Type":"ContainerStarted","Data":"d6446c47bab72b0fc6de393f8c0d634768fea3dd06c09855d4308a276ab2bf0a"} Oct 30 18:48:18.995294 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.995284 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:18.996937 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.996901 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" event={"ID":"23bd45ad-4844-4771-ab04-be34f44f15f2","Type":"ContainerStarted","Data":"94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0"} Oct 30 18:48:18.997068 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.996950 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" event={"ID":"23bd45ad-4844-4771-ab04-be34f44f15f2","Type":"ContainerStarted","Data":"3ac6ed55df7e9172fcc860bbdb8c60c7d50c60f343c7762d23e1ef5b092b53bc"} Oct 30 18:48:18.997133 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:18.997083 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:19.019527 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.019483 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" podStartSLOduration=2.019471139 podStartE2EDuration="2.019471139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:19.017861747 +0000 UTC m=+679.346273132" watchObservedRunningTime="2026-10-30 18:48:19.019471139 +0000 UTC m=+679.347882523" Oct 30 18:48:19.040954 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.040912 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" podStartSLOduration=2.040897074 podStartE2EDuration="2.040897074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:19.039550138 +0000 UTC m=+679.367961523" watchObservedRunningTime="2026-10-30 18:48:19.040897074 +0000 UTC m=+679.369308459" Oct 30 18:48:19.444322 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.444289 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp"] Oct 30 18:48:19.447882 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.447866 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:19.458035 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.458014 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0458df\"" Oct 30 18:48:19.466392 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.466371 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp"] Oct 30 18:48:19.530115 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.530094 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0458df-dockercfg-xmsrg\"" Oct 30 18:48:19.549954 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.549925 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfxw9\" (UniqueName: \"kubernetes.io/projected/7676fec7-190b-4c60-8cbd-aded416ed747-kube-api-access-hfxw9\") pod \"payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp\" (UID: \"7676fec7-190b-4c60-8cbd-aded416ed747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:19.550081 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.549978 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7676fec7-190b-4c60-8cbd-aded416ed747-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp\" (UID: \"7676fec7-190b-4c60-8cbd-aded416ed747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:19.651222 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.651193 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfxw9\" (UniqueName: \"kubernetes.io/projected/7676fec7-190b-4c60-8cbd-aded416ed747-kube-api-access-hfxw9\") pod \"payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp\" (UID: \"7676fec7-190b-4c60-8cbd-aded416ed747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:19.651464 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.651248 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7676fec7-190b-4c60-8cbd-aded416ed747-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp\" (UID: \"7676fec7-190b-4c60-8cbd-aded416ed747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:19.652085 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.652062 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7676fec7-190b-4c60-8cbd-aded416ed747-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp\" (UID: \"7676fec7-190b-4c60-8cbd-aded416ed747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:19.665209 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.665189 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfxw9\" (UniqueName: \"kubernetes.io/projected/7676fec7-190b-4c60-8cbd-aded416ed747-kube-api-access-hfxw9\") pod \"payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp\" (UID: \"7676fec7-190b-4c60-8cbd-aded416ed747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:19.706073 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.706000 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:48:19.761107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:19.761079 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:20.054721 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.054597 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp"] Oct 30 18:48:20.057021 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:20.056998 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7676fec7_190b_4c60_8cbd_aded416ed747.slice/crio-27e085caaa18c0a2c45e58da6dca7c9e7dad4f86e614d02988693ff455a47a7c WatchSource:0}: Error finding container 27e085caaa18c0a2c45e58da6dca7c9e7dad4f86e614d02988693ff455a47a7c: Status 404 returned error can't find the container with id 27e085caaa18c0a2c45e58da6dca7c9e7dad4f86e614d02988693ff455a47a7c Oct 30 18:48:20.264916 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.264890 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x"] Oct 30 18:48:20.268546 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.268531 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:20.275431 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.275408 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-85cb41\"" Oct 30 18:48:20.278210 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.278187 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x"] Oct 30 18:48:20.358211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.358146 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8qf\" (UniqueName: \"kubernetes.io/projected/725eb7d0-4077-4c1d-9a15-158e18f92fdb-kube-api-access-rs8qf\") pod \"payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x\" (UID: \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:20.358333 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.358221 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725eb7d0-4077-4c1d-9a15-158e18f92fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x\" (UID: \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:20.458793 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.458758 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8qf\" (UniqueName: \"kubernetes.io/projected/725eb7d0-4077-4c1d-9a15-158e18f92fdb-kube-api-access-rs8qf\") pod \"payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x\" (UID: \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:20.459204 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.458818 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725eb7d0-4077-4c1d-9a15-158e18f92fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x\" (UID: \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:20.459542 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.459522 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725eb7d0-4077-4c1d-9a15-158e18f92fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x\" (UID: \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:20.467057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.467034 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8qf\" (UniqueName: \"kubernetes.io/projected/725eb7d0-4077-4c1d-9a15-158e18f92fdb-kube-api-access-rs8qf\") pod \"payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x\" (UID: \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:20.583186 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.583156 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:20.857990 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.857962 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j"] Oct 30 18:48:20.862912 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.862885 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x"] Oct 30 18:48:20.863034 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.863007 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:20.868150 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:20.868118 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725eb7d0_4077_4c1d_9a15_158e18f92fdb.slice/crio-cf0be9ba43b6e616be32c31592fcc92bbe58b98a5e1a0ad08aa115b92c17018d WatchSource:0}: Error finding container cf0be9ba43b6e616be32c31592fcc92bbe58b98a5e1a0ad08aa115b92c17018d: Status 404 returned error can't find the container with id cf0be9ba43b6e616be32c31592fcc92bbe58b98a5e1a0ad08aa115b92c17018d Oct 30 18:48:20.874147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.873389 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-39bde3-serving-cert\"" Oct 30 18:48:20.875410 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.875391 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j"] Oct 30 18:48:20.963095 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.963072 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-metrics-certs\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:20.963205 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.963131 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de79c7d-5fbf-4bd1-9916-f696c5a07901-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:20.963281 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.963263 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-maas-api-tls\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:20.963314 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.963299 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ddl\" (UniqueName: \"kubernetes.io/projected/8de79c7d-5fbf-4bd1-9916-f696c5a07901-kube-api-access-59ddl\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:20.974340 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:20.974320 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 18:48:21.018412 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.018372 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" event={"ID":"725eb7d0-4077-4c1d-9a15-158e18f92fdb","Type":"ContainerStarted","Data":"bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab"} Oct 30 18:48:21.018575 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.018462 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" event={"ID":"725eb7d0-4077-4c1d-9a15-158e18f92fdb","Type":"ContainerStarted","Data":"cf0be9ba43b6e616be32c31592fcc92bbe58b98a5e1a0ad08aa115b92c17018d"} Oct 30 18:48:21.018964 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.018934 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:21.020383 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.020354 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" event={"ID":"7676fec7-190b-4c60-8cbd-aded416ed747","Type":"ContainerStarted","Data":"76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606"} Oct 30 18:48:21.020472 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.020405 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" event={"ID":"7676fec7-190b-4c60-8cbd-aded416ed747","Type":"ContainerStarted","Data":"27e085caaa18c0a2c45e58da6dca7c9e7dad4f86e614d02988693ff455a47a7c"} Oct 30 18:48:21.020616 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.020579 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:21.038369 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.038319 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" podStartSLOduration=1.038302842 podStartE2EDuration="1.038302842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:21.03797268 +0000 UTC m=+681.366384066" watchObservedRunningTime="2026-10-30 18:48:21.038302842 +0000 UTC m=+681.366714227" Oct 30 18:48:21.063741 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.063714 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-maas-api-tls\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.063914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.063770 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ddl\" (UniqueName: \"kubernetes.io/projected/8de79c7d-5fbf-4bd1-9916-f696c5a07901-kube-api-access-59ddl\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.063914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.063854 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-metrics-certs\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.064109 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.063954 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de79c7d-5fbf-4bd1-9916-f696c5a07901-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.064631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.064580 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de79c7d-5fbf-4bd1-9916-f696c5a07901-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.067124 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.067093 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-maas-api-tls\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.067255 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.067239 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-metrics-certs\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.076479 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.076452 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ddl\" (UniqueName: \"kubernetes.io/projected/8de79c7d-5fbf-4bd1-9916-f696c5a07901-kube-api-access-59ddl\") pod \"maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.190747 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.190724 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:21.481088 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.480978 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" podStartSLOduration=2.480964396 podStartE2EDuration="2.480964396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:21.06129822 +0000 UTC m=+681.389709604" watchObservedRunningTime="2026-10-30 18:48:21.480964396 +0000 UTC m=+681.809375784" Oct 30 18:48:21.482644 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.482599 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j"] Oct 30 18:48:21.485416 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:21.485385 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de79c7d_5fbf_4bd1_9916_f696c5a07901.slice/crio-cc72096df99df62b80e987710b0982be29acdbf447880d7eb75072a1578fd3a8 WatchSource:0}: Error finding container cc72096df99df62b80e987710b0982be29acdbf447880d7eb75072a1578fd3a8: Status 404 returned error can't find the container with id cc72096df99df62b80e987710b0982be29acdbf447880d7eb75072a1578fd3a8 Oct 30 18:48:21.486827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:21.486810 5959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:48:22.039884 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.039845 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" event={"ID":"8de79c7d-5fbf-4bd1-9916-f696c5a07901","Type":"ContainerStarted","Data":"cc72096df99df62b80e987710b0982be29acdbf447880d7eb75072a1578fd3a8"} Oct 30 18:48:22.445010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.444979 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk"] Oct 30 18:48:22.448995 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.448973 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.457631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.457575 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-85cb41-serving-cert\"" Oct 30 18:48:22.462525 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.462500 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk"] Oct 30 18:48:22.483780 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.483741 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-metrics-certs\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.484136 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.483834 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd779df3-489c-491c-8f35-747a9536e4dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.484136 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.483890 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9qvb\" (UniqueName: \"kubernetes.io/projected/cd779df3-489c-491c-8f35-747a9536e4dd-kube-api-access-h9qvb\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.484136 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.484040 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-maas-api-tls\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.584823 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.584793 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-metrics-certs\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.584981 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.584835 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd779df3-489c-491c-8f35-747a9536e4dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.584981 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.584888 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9qvb\" (UniqueName: \"kubernetes.io/projected/cd779df3-489c-491c-8f35-747a9536e4dd-kube-api-access-h9qvb\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.584981 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.584955 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-maas-api-tls\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.585668 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.585596 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd779df3-489c-491c-8f35-747a9536e4dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.587337 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.587318 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-metrics-certs\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.587498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.587478 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-maas-api-tls\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.592553 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.592533 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9qvb\" (UniqueName: \"kubernetes.io/projected/cd779df3-489c-491c-8f35-747a9536e4dd-kube-api-access-h9qvb\") pod \"maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:22.764283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:22.764194 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:23.037769 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:23.037536 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk"] Oct 30 18:48:23.041198 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:23.041166 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd779df3_489c_491c_8f35_747a9536e4dd.slice/crio-bfffe584936f2310abfecae282d9ce61e2ae25b4e16983eaca9b6d9931ea012d WatchSource:0}: Error finding container bfffe584936f2310abfecae282d9ce61e2ae25b4e16983eaca9b6d9931ea012d: Status 404 returned error can't find the container with id bfffe584936f2310abfecae282d9ce61e2ae25b4e16983eaca9b6d9931ea012d Oct 30 18:48:23.048207 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:23.048182 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" event={"ID":"8de79c7d-5fbf-4bd1-9916-f696c5a07901","Type":"ContainerStarted","Data":"c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312"} Oct 30 18:48:23.069234 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:23.069188 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" podStartSLOduration=2.534656914 podStartE2EDuration="3.069174224s" totalImagesPullingTime="534.51731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:23.068277148 +0000 UTC m=+683.396688533" watchObservedRunningTime="2026-10-30 18:48:23.069174224 +0000 UTC m=+683.397585608" Oct 30 18:48:24.054646 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:24.054596 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" event={"ID":"cd779df3-489c-491c-8f35-747a9536e4dd","Type":"ContainerStarted","Data":"c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4"} Oct 30 18:48:24.055049 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:24.054667 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" event={"ID":"cd779df3-489c-491c-8f35-747a9536e4dd","Type":"ContainerStarted","Data":"bfffe584936f2310abfecae282d9ce61e2ae25b4e16983eaca9b6d9931ea012d"} Oct 30 18:48:24.055049 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:24.054875 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:24.055049 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:24.054908 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:24.077390 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:24.077339 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" podStartSLOduration=1.6855784790000001 podStartE2EDuration="2.077323941s" totalImagesPullingTime="391.745462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:24.075070716 +0000 UTC m=+684.403482102" watchObservedRunningTime="2026-10-30 18:48:24.077323941 +0000 UTC m=+684.405735319" Oct 30 18:48:27.779630 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:27.779569 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:48:28.999498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:28.999472 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 18:48:29.965198 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:29.965164 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:48:30.005228 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:30.005201 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 18:48:30.005756 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:30.005547 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 18:48:30.066082 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:30.066053 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 18:48:30.066616 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:30.066577 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 18:48:32.043240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:32.043215 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 18:48:33.050866 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:33.050836 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 18:48:40.792742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:40.792702 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:48:42.880118 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.880074 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9"] Oct 30 18:48:42.892742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.892714 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:42.894844 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.894815 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9"] Oct 30 18:48:42.901156 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.901135 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9de872\"/\"default-dockercfg-ddmfm\"" Oct 30 18:48:42.901325 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.901248 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9de872\"/\"simulator-w0-9de872-kserve-self-signed-certs\"" Oct 30 18:48:42.905629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.905581 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9de872\"/\"kube-root-ca.crt\"" Oct 30 18:48:42.905719 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.905652 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9de872\"/\"openshift-service-ca.crt\"" Oct 30 18:48:42.982778 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.982740 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-home\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:42.982956 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.982792 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tmp-dir\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:42.982956 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.982901 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tls-certs\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:42.983076 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.982953 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-dshm\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:42.983137 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.983070 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-model-cache\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:42.983197 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:42.983126 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6tt\" (UniqueName: \"kubernetes.io/projected/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-kube-api-access-2n6tt\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.084746 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.084713 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-dshm\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.084946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.084756 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-model-cache\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.084946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.084782 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6tt\" (UniqueName: \"kubernetes.io/projected/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-kube-api-access-2n6tt\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.084946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.084834 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-home\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.084946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.084851 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tmp-dir\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.084946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.084878 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tls-certs\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.085229 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.085203 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-model-cache\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.085285 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.085243 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-home\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.085324 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.085280 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tmp-dir\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.087321 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.087299 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-dshm\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.087492 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.087474 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tls-certs\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.096916 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.096895 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6tt\" (UniqueName: \"kubernetes.io/projected/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-kube-api-access-2n6tt\") pod \"simulator-w0-9de872-kserve-75bb5f55fb-4bln9\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.210788 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.210755 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:43.490124 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:43.489981 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9"] Oct 30 18:48:43.492931 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:43.492904 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cb88e37_7ad7_4e05_b2af_f2571e01a2d2.slice/crio-1837ac92b4d7cf795da2db388d990843b06b711117054971b1f33d4ad49818b4 WatchSource:0}: Error finding container 1837ac92b4d7cf795da2db388d990843b06b711117054971b1f33d4ad49818b4: Status 404 returned error can't find the container with id 1837ac92b4d7cf795da2db388d990843b06b711117054971b1f33d4ad49818b4 Oct 30 18:48:44.167477 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:44.167421 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" event={"ID":"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2","Type":"ContainerStarted","Data":"1837ac92b4d7cf795da2db388d990843b06b711117054971b1f33d4ad49818b4"} Oct 30 18:48:45.176557 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.176526 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml"] Oct 30 18:48:45.180318 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.180293 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" event={"ID":"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2","Type":"ContainerStarted","Data":"36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb"} Oct 30 18:48:45.180451 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.180369 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.187532 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.187492 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0458df\"/\"kube-root-ca.crt\"" Oct 30 18:48:45.187832 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.187724 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0458df\"/\"openshift-service-ca.crt\"" Oct 30 18:48:45.188685 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.188627 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0458df\"/\"default-dockercfg-ltv6n\"" Oct 30 18:48:45.188792 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.188719 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0458df\"/\"simulator-w4-0458df-kserve-self-signed-certs\"" Oct 30 18:48:45.191858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.191833 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml"] Oct 30 18:48:45.205186 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.205124 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podStartSLOduration=1.9787915360000001 podStartE2EDuration="3.205104736s" totalImagesPullingTime="1.2263132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:45.201735723 +0000 UTC m=+705.530147110" watchObservedRunningTime="2026-10-30 18:48:45.205104736 +0000 UTC m=+705.533516121" Oct 30 18:48:45.211275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.211253 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:45.308987 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.308947 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-home\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.309179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.309005 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-dshm\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.309179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.309025 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-tmp-dir\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.309290 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.309191 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hg86\" (UniqueName: \"kubernetes.io/projected/36cc2519-050e-4689-975a-b33a9dd76d5e-kube-api-access-6hg86\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.309364 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.309338 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cc2519-050e-4689-975a-b33a9dd76d5e-tls-certs\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.309457 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.309440 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-model-cache\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410294 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410253 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-model-cache\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410466 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410314 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-home\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410466 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410353 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-dshm\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410466 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410376 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-tmp-dir\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410466 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410410 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hg86\" (UniqueName: \"kubernetes.io/projected/36cc2519-050e-4689-975a-b33a9dd76d5e-kube-api-access-6hg86\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410700 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410472 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cc2519-050e-4689-975a-b33a9dd76d5e-tls-certs\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410792 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410769 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-model-cache\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410862 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410790 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-tmp-dir\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.410862 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.410839 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-home\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.412928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.412909 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-dshm\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.413182 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.413161 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cc2519-050e-4689-975a-b33a9dd76d5e-tls-certs\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.418932 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.418906 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hg86\" (UniqueName: \"kubernetes.io/projected/36cc2519-050e-4689-975a-b33a9dd76d5e-kube-api-access-6hg86\") pod \"simulator-w4-0458df-kserve-6885b74b94-8g6ml\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.497329 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.497236 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:45.773151 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:45.773084 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml"] Oct 30 18:48:45.775496 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:45.775470 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36cc2519_050e_4689_975a_b33a9dd76d5e.slice/crio-b50035723a7a3a3b997bdc396c8ec9ca70a49db60dc7bed02f0b6e786c65b0cd WatchSource:0}: Error finding container b50035723a7a3a3b997bdc396c8ec9ca70a49db60dc7bed02f0b6e786c65b0cd: Status 404 returned error can't find the container with id b50035723a7a3a3b997bdc396c8ec9ca70a49db60dc7bed02f0b6e786c65b0cd Oct 30 18:48:46.185283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.185246 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" event={"ID":"36cc2519-050e-4689-975a-b33a9dd76d5e","Type":"ContainerStarted","Data":"ef911e14d3567f948ea24ece53e438168b7086ad3894dd257deb32a6e4bda589"} Oct 30 18:48:46.186117 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.185298 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" event={"ID":"36cc2519-050e-4689-975a-b33a9dd76d5e","Type":"ContainerStarted","Data":"b50035723a7a3a3b997bdc396c8ec9ca70a49db60dc7bed02f0b6e786c65b0cd"} Oct 30 18:48:46.187720 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.187698 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98"] Oct 30 18:48:46.191455 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.191438 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.202113 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.202087 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"kube-root-ca.crt\"" Oct 30 18:48:46.202277 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.202263 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"openshift-service-ca.crt\"" Oct 30 18:48:46.202371 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.202358 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"default-dockercfg-pfnh7\"" Oct 30 18:48:46.203146 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.203124 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"simulator-w1-85cb41-kserve-self-signed-certs\"" Oct 30 18:48:46.203799 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.203778 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98"] Oct 30 18:48:46.213720 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.213620 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podStartSLOduration=1.213582852 podStartE2EDuration="1.213582852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:46.211862234 +0000 UTC m=+706.540273614" watchObservedRunningTime="2026-10-30 18:48:46.213582852 +0000 UTC m=+706.541994239" Oct 30 18:48:46.320463 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.320418 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-dshm\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.320690 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.320575 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-tmp-dir\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.320764 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.320707 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3860e2-becb-4660-9d93-ff8c50373fb7-tls-certs\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.320827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.320764 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-home\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.320827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.320798 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvtdf\" (UniqueName: \"kubernetes.io/projected/1f3860e2-becb-4660-9d93-ff8c50373fb7-kube-api-access-dvtdf\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.320933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.320908 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-model-cache\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.422468 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.422436 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-tmp-dir\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.422655 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.422477 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3860e2-becb-4660-9d93-ff8c50373fb7-tls-certs\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.422655 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.422598 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-home\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.422763 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.422660 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvtdf\" (UniqueName: \"kubernetes.io/projected/1f3860e2-becb-4660-9d93-ff8c50373fb7-kube-api-access-dvtdf\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.422763 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.422699 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-model-cache\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.422861 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.422775 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-dshm\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.422932 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.422906 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-tmp-dir\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.422992 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.422935 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-home\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.423108 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.423088 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-model-cache\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.425072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.425055 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-dshm\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.425144 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.425120 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3860e2-becb-4660-9d93-ff8c50373fb7-tls-certs\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.431031 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.431006 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvtdf\" (UniqueName: \"kubernetes.io/projected/1f3860e2-becb-4660-9d93-ff8c50373fb7-kube-api-access-dvtdf\") pod \"simulator-w1-85cb41-kserve-7c986d86bc-6gd98\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.498070 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.497994 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:46.508270 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.508232 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:46.794653 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:46.788899 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98"] Oct 30 18:48:46.797928 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:48:46.797897 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f3860e2_becb_4660_9d93_ff8c50373fb7.slice/crio-fbbd59b5a3d5140860a9549897a1c4c604c676c35aea42b4eca45f9798a1fe17 WatchSource:0}: Error finding container fbbd59b5a3d5140860a9549897a1c4c604c676c35aea42b4eca45f9798a1fe17: Status 404 returned error can't find the container with id fbbd59b5a3d5140860a9549897a1c4c604c676c35aea42b4eca45f9798a1fe17 Oct 30 18:48:47.192348 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:47.192302 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" event={"ID":"1f3860e2-becb-4660-9d93-ff8c50373fb7","Type":"ContainerStarted","Data":"ec8f48c7d9fb12061d5d9492a8e5f78e81d5dae2942226af437d25385ec49ad9"} Oct 30 18:48:47.192348 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:47.192351 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" event={"ID":"1f3860e2-becb-4660-9d93-ff8c50373fb7","Type":"ContainerStarted","Data":"fbbd59b5a3d5140860a9549897a1c4c604c676c35aea42b4eca45f9798a1fe17"} Oct 30 18:48:47.211183 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:47.211143 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podStartSLOduration=1.211128426 podStartE2EDuration="1.211128426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:47.209300461 +0000 UTC m=+707.537711846" watchObservedRunningTime="2026-10-30 18:48:47.211128426 +0000 UTC m=+707.539539849" Oct 30 18:48:47.509026 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:47.508948 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:49.333770 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:49.333736 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:48:53.211426 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:53.211382 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:53.225024 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:53.225000 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:53.236351 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:53.236327 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:48:55.497676 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:55.497636 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:55.510246 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:55.510222 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:56.253353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:56.253327 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:48:56.508594 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:56.508509 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:56.526717 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:56.526688 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:57.257775 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:57.257749 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:48:57.780691 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:48:57.780644 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:48:59.747204 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:48:59.747167 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:49:06.286105 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.286072 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm"] Oct 30 18:49:06.291762 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.291742 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.323487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.323456 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-home\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.323646 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.323533 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0d64dc-513a-4110-9364-29c7960c74a3-tls-certs\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.323646 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.323628 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-tmp-dir\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.323725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.323649 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvrb\" (UniqueName: \"kubernetes.io/projected/cd0d64dc-513a-4110-9364-29c7960c74a3-kube-api-access-kkvrb\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.323725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.323675 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-dshm\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.323791 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.323751 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-model-cache\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.333127 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.333101 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0458df\"/\"premium-w4-0458df-kserve-self-signed-certs\"" Oct 30 18:49:06.333635 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.333594 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm"] Oct 30 18:49:06.425299 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.425261 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-tmp-dir\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.425505 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.425311 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvrb\" (UniqueName: \"kubernetes.io/projected/cd0d64dc-513a-4110-9364-29c7960c74a3-kube-api-access-kkvrb\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.425505 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.425486 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-dshm\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.425662 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.425546 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-model-cache\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.425720 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.425661 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-home\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.425775 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.425766 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0d64dc-513a-4110-9364-29c7960c74a3-tls-certs\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.425834 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.425779 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-tmp-dir\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.425969 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.425947 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-model-cache\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.426103 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.426081 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-home\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.428232 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.428207 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0d64dc-513a-4110-9364-29c7960c74a3-tls-certs\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.428330 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.428291 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-dshm\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.434510 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.434482 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvrb\" (UniqueName: \"kubernetes.io/projected/cd0d64dc-513a-4110-9364-29c7960c74a3-kube-api-access-kkvrb\") pod \"premium-w4-0458df-kserve-5d48c994f9-9t6wm\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.603878 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.603803 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:06.943629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:06.943568 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm"] Oct 30 18:49:06.948106 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:49:06.948078 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd0d64dc_513a_4110_9364_29c7960c74a3.slice/crio-88c25b21cef4223dabfc83c5aa2492b15d2c159b70a0c47361c9fc4264db926d WatchSource:0}: Error finding container 88c25b21cef4223dabfc83c5aa2492b15d2c159b70a0c47361c9fc4264db926d: Status 404 returned error can't find the container with id 88c25b21cef4223dabfc83c5aa2492b15d2c159b70a0c47361c9fc4264db926d Oct 30 18:49:07.307835 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:07.307738 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" event={"ID":"cd0d64dc-513a-4110-9364-29c7960c74a3","Type":"ContainerStarted","Data":"cdca4ce44010bff7282bf11cf2720f8cc09a89df69f52261dc9733cea8bf8c8e"} Oct 30 18:49:07.307835 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:07.307785 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" event={"ID":"cd0d64dc-513a-4110-9364-29c7960c74a3","Type":"ContainerStarted","Data":"88c25b21cef4223dabfc83c5aa2492b15d2c159b70a0c47361c9fc4264db926d"} Oct 30 18:49:07.334237 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:07.334186 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podStartSLOduration=1.334172745 podStartE2EDuration="1.334172745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:07.33172075 +0000 UTC m=+727.660132146" watchObservedRunningTime="2026-10-30 18:49:07.334172745 +0000 UTC m=+727.662584131" Oct 30 18:49:07.604446 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:07.604366 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:14.817783 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:14.817700 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:49:16.604766 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:16.604729 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:16.617486 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:16.617463 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:17.378023 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:17.377985 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:49:27.780103 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:49:27.780060 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:49:33.737352 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:33.737292 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:49:42.036445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.036411 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l"] Oct 30 18:49:42.042151 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.042132 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.049989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.049794 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-39bde3-openshift-default-dockercfg-vwm86\"" Oct 30 18:49:42.050628 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.050588 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l"] Oct 30 18:49:42.083894 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.083868 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e0824e-05df-42c1-a737-90283fecef74-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.084078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.083919 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.084078 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.084008 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.084169 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.084106 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.084169 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.084154 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.084264 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.084192 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.084323 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.084255 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e0824e-05df-42c1-a737-90283fecef74-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.084391 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.084355 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxwt\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-kube-api-access-crxwt\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.084440 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.084402 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.184946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.184903 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.184946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.184948 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185183 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185058 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185183 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185117 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185297 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185184 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185297 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185209 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e0824e-05df-42c1-a737-90283fecef74-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185297 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185262 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxwt\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-kube-api-access-crxwt\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185297 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185268 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185319 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185334 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185390 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185422 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e0824e-05df-42c1-a737-90283fecef74-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185428 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.185905 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.185882 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e0824e-05df-42c1-a737-90283fecef74-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.187559 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.187542 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.187840 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.187824 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e0824e-05df-42c1-a737-90283fecef74-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.192640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.192593 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxwt\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-kube-api-access-crxwt\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.192934 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.192915 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.358247 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.358156 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:42.638926 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.638901 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l"] Oct 30 18:49:42.641572 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:49:42.641541 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e0824e_05df_42c1_a737_90283fecef74.slice/crio-474cb89da8bd09b1acc5d51c50c409c99f5f8aec336cfd42019bb8b305133f75 WatchSource:0}: Error finding container 474cb89da8bd09b1acc5d51c50c409c99f5f8aec336cfd42019bb8b305133f75: Status 404 returned error can't find the container with id 474cb89da8bd09b1acc5d51c50c409c99f5f8aec336cfd42019bb8b305133f75 Oct 30 18:49:42.643855 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.643825 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:49:42.643926 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.643887 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:49:42.643926 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:42.643917 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:49:43.493738 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:43.493701 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" event={"ID":"89e0824e-05df-42c1-a737-90283fecef74","Type":"ContainerStarted","Data":"ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8"} Oct 30 18:49:43.494112 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:43.493776 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" event={"ID":"89e0824e-05df-42c1-a737-90283fecef74","Type":"ContainerStarted","Data":"474cb89da8bd09b1acc5d51c50c409c99f5f8aec336cfd42019bb8b305133f75"} Oct 30 18:49:43.537487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:43.537416 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" podStartSLOduration=1.537397685 podStartE2EDuration="1.537397685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:43.534314069 +0000 UTC m=+763.862725452" watchObservedRunningTime="2026-10-30 18:49:43.537397685 +0000 UTC m=+763.865809070" Oct 30 18:49:44.358773 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:44.358741 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:44.360399 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:44.360379 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:49:44.360460 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:44.360429 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:49:45.359420 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:45.359385 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:49:45.359865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:45.359439 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:49:46.359340 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:46.359306 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:49:46.359633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:46.359363 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:49:47.358902 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:47.358870 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:49:47.359082 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:47.358924 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:49:48.359561 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:48.359521 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:49:48.359962 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:48.359587 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:49:49.755688 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:49.755549 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:49.756341 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:49.755956 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:49.757504 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:49:49.757480 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 18:49:57.778366 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:49:57.778326 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:50:09.758447 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.758405 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9"] Oct 30 18:50:09.765705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.765679 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:09.776430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.776403 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9"] Oct 30 18:50:09.777405 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.777371 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"default-dockercfg-pkwqf\"" Oct 30 18:50:09.777405 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.777391 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"e2e-conflict-a-8eedd2a0-kserve-self-signed-certs\"" Oct 30 18:50:09.777405 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.777383 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"kube-root-ca.crt\"" Oct 30 18:50:09.777584 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.777442 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"openshift-service-ca.crt\"" Oct 30 18:50:09.899552 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.899513 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-model-cache\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:09.899771 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.899571 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-dshm\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:09.899771 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.899710 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6md9n\" (UniqueName: \"kubernetes.io/projected/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-kube-api-access-6md9n\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:09.899771 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.899758 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-home\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:09.899898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.899802 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tls-certs\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:09.899972 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:09.899944 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tmp-dir\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.001274 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001233 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-model-cache\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.001439 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001327 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-dshm\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.001439 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001365 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6md9n\" (UniqueName: \"kubernetes.io/projected/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-kube-api-access-6md9n\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.001439 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001407 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-home\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.001566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001451 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tls-certs\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.001566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001504 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tmp-dir\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.001724 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001699 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-model-cache\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.001888 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001865 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tmp-dir\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.002034 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.001803 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-home\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.003987 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.003966 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-dshm\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.004593 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.004571 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tls-certs\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.015444 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.015390 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6md9n\" (UniqueName: \"kubernetes.io/projected/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-kube-api-access-6md9n\") pod \"e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.078652 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.078589 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:10.371274 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.371161 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9"] Oct 30 18:50:10.375618 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:50:10.375575 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bd4aa33_fe70_4078_9fdc_4be2c2dd9c4e.slice/crio-ec256363960dc580e721ae61a4af8eb7999beea6e4691ef615a7185da72c8aaf WatchSource:0}: Error finding container ec256363960dc580e721ae61a4af8eb7999beea6e4691ef615a7185da72c8aaf: Status 404 returned error can't find the container with id ec256363960dc580e721ae61a4af8eb7999beea6e4691ef615a7185da72c8aaf Oct 30 18:50:10.658873 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.658839 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" event={"ID":"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e","Type":"ContainerStarted","Data":"810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b"} Oct 30 18:50:10.659063 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.658880 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" event={"ID":"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e","Type":"ContainerStarted","Data":"ec256363960dc580e721ae61a4af8eb7999beea6e4691ef615a7185da72c8aaf"} Oct 30 18:50:10.684166 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:10.684113 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" podStartSLOduration=1.684099927 podStartE2EDuration="1.684099927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:10.681941343 +0000 UTC m=+791.010352727" watchObservedRunningTime="2026-10-30 18:50:10.684099927 +0000 UTC m=+791.012511312" Oct 30 18:50:11.079022 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:11.078929 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:13.788761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:13.788722 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:50:18.611814 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.611787 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm"] Oct 30 18:50:18.615733 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.615711 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.626990 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.626449 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0458df\"/\"distinct-w4-0458df-kserve-self-signed-certs\"" Oct 30 18:50:18.633663 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.628344 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm"] Oct 30 18:50:18.792252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.792206 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ae7e50-6451-4b7e-8691-999aa3234d94-tls-certs\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.792484 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.792264 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-tmp-dir\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.792484 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.792379 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldg2b\" (UniqueName: \"kubernetes.io/projected/28ae7e50-6451-4b7e-8691-999aa3234d94-kube-api-access-ldg2b\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.792484 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.792401 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-model-cache\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.792484 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.792451 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-dshm\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.792730 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.792568 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-home\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894014 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.893978 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-model-cache\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894213 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.894061 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-dshm\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894213 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.894109 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-home\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894331 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.894249 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ae7e50-6451-4b7e-8691-999aa3234d94-tls-certs\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894457 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.894436 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-model-cache\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894577 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.894518 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-home\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894675 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.894599 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-tmp-dir\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894675 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.894656 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldg2b\" (UniqueName: \"kubernetes.io/projected/28ae7e50-6451-4b7e-8691-999aa3234d94-kube-api-access-ldg2b\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.894900 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.894881 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-tmp-dir\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.896835 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.896806 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-dshm\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.897060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.897043 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ae7e50-6451-4b7e-8691-999aa3234d94-tls-certs\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.903595 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.903572 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldg2b\" (UniqueName: \"kubernetes.io/projected/28ae7e50-6451-4b7e-8691-999aa3234d94-kube-api-access-ldg2b\") pod \"distinct-w4-0458df-kserve-c586879bb-r6wlm\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:18.938885 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:18.938848 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:19.220179 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:19.220154 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm"] Oct 30 18:50:19.222896 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:50:19.222866 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28ae7e50_6451_4b7e_8691_999aa3234d94.slice/crio-02f7ab40f7436f19fd071368fff39df8e0accd795cfc76a40bfd641b1207e42c WatchSource:0}: Error finding container 02f7ab40f7436f19fd071368fff39df8e0accd795cfc76a40bfd641b1207e42c: Status 404 returned error can't find the container with id 02f7ab40f7436f19fd071368fff39df8e0accd795cfc76a40bfd641b1207e42c Oct 30 18:50:19.707989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:19.707932 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" event={"ID":"28ae7e50-6451-4b7e-8691-999aa3234d94","Type":"ContainerStarted","Data":"7620fecbc327c416e6d96a2e32d80659c3fdb807dc39e88187e9f0460ab7274f"} Oct 30 18:50:19.708365 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:19.707996 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" event={"ID":"28ae7e50-6451-4b7e-8691-999aa3234d94","Type":"ContainerStarted","Data":"02f7ab40f7436f19fd071368fff39df8e0accd795cfc76a40bfd641b1207e42c"} Oct 30 18:50:19.729423 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:19.729375 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podStartSLOduration=1.729359642 podStartE2EDuration="1.729359642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:19.727420488 +0000 UTC m=+800.055831873" watchObservedRunningTime="2026-10-30 18:50:19.729359642 +0000 UTC m=+800.057771025" Oct 30 18:50:19.939395 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:19.939363 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:20.079182 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:20.079088 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:20.091507 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:20.091485 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:20.724909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:20.724884 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:50:22.636732 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.636705 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4"] Oct 30 18:50:22.640735 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.640713 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.650657 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.650636 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2889dd-openshift-default-dockercfg-qgr7b\"" Oct 30 18:50:22.651571 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.651546 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4"] Oct 30 18:50:22.736217 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736186 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.736412 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736228 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.736412 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736259 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.736412 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736315 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.736555 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736440 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlf9r\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-kube-api-access-xlf9r\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.736555 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736477 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.736660 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736580 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2606db6-8b96-4360-9519-6cda6ca1380f-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.736660 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736600 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.736660 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.736649 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837369 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837336 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837558 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837392 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2606db6-8b96-4360-9519-6cda6ca1380f-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837558 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837408 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837558 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837422 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837558 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837525 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837803 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837554 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837857 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837832 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837864 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.837911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837880 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.838009 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837915 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlf9r\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-kube-api-access-xlf9r\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.838009 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.837987 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.838129 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.838108 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.838257 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.838233 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.838545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.838526 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2606db6-8b96-4360-9519-6cda6ca1380f-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.840097 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.840077 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.840215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.840199 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.845749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.845726 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.845749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.845740 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlf9r\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-kube-api-access-xlf9r\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:22.955125 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:22.955044 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:23.240092 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:23.239901 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4"] Oct 30 18:50:23.252496 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:23.248431 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:23.252496 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:23.248624 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:23.252496 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:23.248784 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:23.731785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:23.731749 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" event={"ID":"a2606db6-8b96-4360-9519-6cda6ca1380f","Type":"ContainerStarted","Data":"c0d4f8b267ad7e800e72f8d65d5243ab7dcda505bed6af99b09b99aa44fe2463"} Oct 30 18:50:23.732209 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:23.731802 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" event={"ID":"a2606db6-8b96-4360-9519-6cda6ca1380f","Type":"ContainerStarted","Data":"0a613ebd3f1dd15747e3590a54659c684f8bc1f9a7eea12caa2dec32b53cdc7e"} Oct 30 18:50:23.754982 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:23.754933 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podStartSLOduration=1.754918128 podStartE2EDuration="1.754918128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:23.751890774 +0000 UTC m=+804.080302161" watchObservedRunningTime="2026-10-30 18:50:23.754918128 +0000 UTC m=+804.083329511" Oct 30 18:50:23.955772 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:23.955725 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:24.956976 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:24.956943 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:50:24.957348 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:24.957007 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 18:50:25.956082 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:25.956045 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:50:25.956280 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:25.956118 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 18:50:26.956405 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:26.956346 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:50:26.956923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:26.956431 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 18:50:27.106956 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.106921 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7"] Oct 30 18:50:27.118977 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.118945 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.141860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.141823 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7"] Oct 30 18:50:27.185325 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185289 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.185511 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185333 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.185511 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185364 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.185645 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185509 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.185645 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185540 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrmn\" (UniqueName: \"kubernetes.io/projected/bf0d5ca3-7ba8-481c-b29d-702244848bbf-kube-api-access-xsrmn\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.185733 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185655 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.185768 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185730 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.185862 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185845 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.185923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.185897 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.287500 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287405 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.287500 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287460 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.287500 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287498 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.287809 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287537 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.287809 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287568 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrmn\" (UniqueName: \"kubernetes.io/projected/bf0d5ca3-7ba8-481c-b29d-702244848bbf-kube-api-access-xsrmn\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.287809 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287637 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.287809 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287673 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.287809 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287791 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.288074 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287838 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.288074 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.287868 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.288245 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.288220 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.288309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.288276 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.288309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.288276 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.288445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.288420 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.289939 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.289910 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.290299 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.290280 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.299338 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.299260 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrmn\" (UniqueName: \"kubernetes.io/projected/bf0d5ca3-7ba8-481c-b29d-702244848bbf-kube-api-access-xsrmn\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.314631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.303041 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf0d5ca3-7ba8-481c-b29d-702244848bbf-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-bkpg7\" (UID: \"bf0d5ca3-7ba8-481c-b29d-702244848bbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.458278 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.458238 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:27.783871 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:50:27.783827 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:50:27.793370 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:50:27.793334 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0d5ca3_7ba8_481c_b29d_702244848bbf.slice/crio-aa295797b3677d0dfba368c3d45314d1759e80a9beba074591b73b7ce44be835 WatchSource:0}: Error finding container aa295797b3677d0dfba368c3d45314d1759e80a9beba074591b73b7ce44be835: Status 404 returned error can't find the container with id aa295797b3677d0dfba368c3d45314d1759e80a9beba074591b73b7ce44be835 Oct 30 18:50:27.810820 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.795940 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:27.810820 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.796030 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:27.810820 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.796071 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:27.810820 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.801326 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7"] Oct 30 18:50:27.957634 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.956015 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:50:27.957634 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:27.956077 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 18:50:28.777630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.776125 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" event={"ID":"bf0d5ca3-7ba8-481c-b29d-702244848bbf","Type":"ContainerStarted","Data":"e517624fd14e1c108af607a10ebc8fddc91aa1a4d7cb33e3b3bd5af5ae1bebf4"} Oct 30 18:50:28.777630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.776230 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" event={"ID":"bf0d5ca3-7ba8-481c-b29d-702244848bbf","Type":"ContainerStarted","Data":"aa295797b3677d0dfba368c3d45314d1759e80a9beba074591b73b7ce44be835"} Oct 30 18:50:28.934626 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.934262 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podStartSLOduration=1.93423938 podStartE2EDuration="1.93423938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:28.803034697 +0000 UTC m=+809.131446081" watchObservedRunningTime="2026-10-30 18:50:28.93423938 +0000 UTC m=+809.262650762" Oct 30 18:50:28.939635 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.936619 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq"] Oct 30 18:50:28.962680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.958003 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:28.962680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.958885 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:28.962680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.960759 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:50:28.962680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.960812 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 18:50:28.965176 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.965149 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq"] Oct 30 18:50:28.974186 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.974142 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0458df-openshift-default-dockercfg-2vrss\"" Oct 30 18:50:28.976818 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:28.975915 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044305 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044448 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044497 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044558 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044593 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044657 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044699 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044732 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkd69\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-kube-api-access-vkd69\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.053761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.044766 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.146628 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146489 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.146919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146640 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.146919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146694 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.146919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146754 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.146919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146804 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.146919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146828 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.146919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146881 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.146919 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146913 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkd69\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-kube-api-access-vkd69\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.147293 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146958 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.147293 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.146967 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.147625 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.147533 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.147807 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.147733 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.148450 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.148413 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.148729 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.148699 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.150364 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.150345 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.150505 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.150486 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.156759 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.156720 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkd69\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-kube-api-access-vkd69\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.157246 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.157227 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.284872 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.284835 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:29.458865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.458563 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:29.461766 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.461732 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 18:50:29.461953 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.461793 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podUID="bf0d5ca3-7ba8-481c-b29d-702244848bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 18:50:29.651401 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.651360 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq"] Oct 30 18:50:29.655104 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:50:29.655070 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e4ec7d_7b73_42cc_94d3_5eb46d09872f.slice/crio-c5e3a1bbadd7c4924073d1bbdb366e4974f808191f6e03bd23794da074574ce1 WatchSource:0}: Error finding container c5e3a1bbadd7c4924073d1bbdb366e4974f808191f6e03bd23794da074574ce1: Status 404 returned error can't find the container with id c5e3a1bbadd7c4924073d1bbdb366e4974f808191f6e03bd23794da074574ce1 Oct 30 18:50:29.658214 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.658170 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:29.658306 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.658268 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:29.658350 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.658312 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:29.789228 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.789177 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" event={"ID":"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f","Type":"ContainerStarted","Data":"6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208"} Oct 30 18:50:29.805705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.805667 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" event={"ID":"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f","Type":"ContainerStarted","Data":"c5e3a1bbadd7c4924073d1bbdb366e4974f808191f6e03bd23794da074574ce1"} Oct 30 18:50:29.843626 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.843578 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:50:29.863509 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.863440 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podStartSLOduration=1.863427331 podStartE2EDuration="1.863427331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:29.833153023 +0000 UTC m=+810.161564410" watchObservedRunningTime="2026-10-30 18:50:29.863427331 +0000 UTC m=+810.191838716" Oct 30 18:50:29.956061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.955984 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 18:50:29.956256 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:29.956234 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 18:50:30.285314 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:30.285230 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:30.286735 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:30.286708 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:50:30.286864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:30.286760 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:50:30.459090 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:30.459035 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 18:50:30.459362 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:30.459107 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podUID="bf0d5ca3-7ba8-481c-b29d-702244848bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 18:50:31.285489 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:31.285461 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:50:31.285948 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:31.285673 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:50:31.413628 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:31.412966 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:31.418632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:31.414679 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:31.437633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:31.424018 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 18:50:31.460823 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:31.460509 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 18:50:31.460823 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:31.460556 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podUID="bf0d5ca3-7ba8-481c-b29d-702244848bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 18:50:32.285301 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:32.285265 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:50:32.285492 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:32.285319 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:50:32.459239 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:32.459193 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 18:50:32.459751 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:32.459254 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podUID="bf0d5ca3-7ba8-481c-b29d-702244848bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 18:50:33.285930 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:33.285895 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:50:33.286080 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:33.285952 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:50:33.459075 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:33.459032 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 18:50:33.459296 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:33.459090 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podUID="bf0d5ca3-7ba8-481c-b29d-702244848bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 18:50:34.285981 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:34.285935 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:50:34.286203 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:34.285992 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:50:34.458901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:34.458865 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 18:50:34.459123 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:34.458923 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podUID="bf0d5ca3-7ba8-481c-b29d-702244848bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 18:50:35.285581 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:35.285544 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:50:35.286111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:35.285624 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:50:35.458874 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:35.458839 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 18:50:35.459083 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:35.458922 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podUID="bf0d5ca3-7ba8-481c-b29d-702244848bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 18:50:36.285736 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:36.285684 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:50:36.286251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:36.285764 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:50:36.992693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:36.992655 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:36.994000 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:36.993937 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:36.996963 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:36.996940 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" Oct 30 18:50:37.289833 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:37.289765 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:37.290201 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:37.290056 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:37.290739 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:37.290723 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 18:50:40.047140 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:40.047109 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:50:42.101765 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.101731 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v"] Oct 30 18:50:42.106459 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.106434 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.121928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.121904 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v"] Oct 30 18:50:42.221121 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221084 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b663c4-e4ce-433f-80b6-52fe26e9f976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.221121 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221134 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.221331 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221223 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.221331 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221280 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.221331 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221308 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.221424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221336 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.221424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221353 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.221500 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221486 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qc4\" (UniqueName: \"kubernetes.io/projected/52b663c4-e4ce-433f-80b6-52fe26e9f976-kube-api-access-j7qc4\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.221566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.221552 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322415 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322376 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322415 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322421 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b663c4-e4ce-433f-80b6-52fe26e9f976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322687 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322444 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322687 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322472 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322687 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322493 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322687 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322516 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322687 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322538 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322687 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322554 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.322687 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322669 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qc4\" (UniqueName: \"kubernetes.io/projected/52b663c4-e4ce-433f-80b6-52fe26e9f976-kube-api-access-j7qc4\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.323101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322831 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.323101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.322911 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.323190 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.323120 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.323190 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.323147 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.323751 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.323721 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b663c4-e4ce-433f-80b6-52fe26e9f976-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.325116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.325092 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.325317 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.325299 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.330927 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.330903 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b663c4-e4ce-433f-80b6-52fe26e9f976-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.331124 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.331102 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qc4\" (UniqueName: \"kubernetes.io/projected/52b663c4-e4ce-433f-80b6-52fe26e9f976-kube-api-access-j7qc4\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-q656v\" (UID: \"52b663c4-e4ce-433f-80b6-52fe26e9f976\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.425778 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.425741 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:42.712408 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.712370 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v"] Oct 30 18:50:42.714118 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:50:42.714091 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b663c4_e4ce_433f_80b6_52fe26e9f976.slice/crio-f3a077282256d14d5024ab3a81052745a55ecf1abaa75514d1622aed6affb62c WatchSource:0}: Error finding container f3a077282256d14d5024ab3a81052745a55ecf1abaa75514d1622aed6affb62c: Status 404 returned error can't find the container with id f3a077282256d14d5024ab3a81052745a55ecf1abaa75514d1622aed6affb62c Oct 30 18:50:42.716834 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.716799 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:42.716940 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.716884 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:42.716940 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.716929 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:50:42.958866 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.958825 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" event={"ID":"52b663c4-e4ce-433f-80b6-52fe26e9f976","Type":"ContainerStarted","Data":"9bfc3fff7f8cbbd801baf3e25f713589fb6c9a054c7943fbd54c935e444ef9b7"} Oct 30 18:50:42.959036 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.958898 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" event={"ID":"52b663c4-e4ce-433f-80b6-52fe26e9f976","Type":"ContainerStarted","Data":"f3a077282256d14d5024ab3a81052745a55ecf1abaa75514d1622aed6affb62c"} Oct 30 18:50:42.981910 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:42.981784 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podStartSLOduration=0.981763484 podStartE2EDuration="981.763484ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:42.979814134 +0000 UTC m=+823.308225535" watchObservedRunningTime="2026-10-30 18:50:42.981763484 +0000 UTC m=+823.310174870" Oct 30 18:50:43.426292 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:43.426249 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:43.428087 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:43.428064 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-q656v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:50:43.428140 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:43.428113 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podUID="52b663c4-e4ce-433f-80b6-52fe26e9f976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:50:44.427170 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:44.427122 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-q656v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:50:44.427859 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:44.427193 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podUID="52b663c4-e4ce-433f-80b6-52fe26e9f976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:50:45.427245 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:45.427198 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-q656v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:50:45.427732 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:45.427269 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podUID="52b663c4-e4ce-433f-80b6-52fe26e9f976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:50:46.426880 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:46.426848 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-q656v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:50:46.427052 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:46.426902 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podUID="52b663c4-e4ce-433f-80b6-52fe26e9f976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:50:47.426517 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:47.426447 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-q656v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:50:47.427068 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:47.426541 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podUID="52b663c4-e4ce-433f-80b6-52fe26e9f976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:50:48.427001 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:48.426962 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-q656v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:50:48.427428 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:48.427013 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podUID="52b663c4-e4ce-433f-80b6-52fe26e9f976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:50:49.426872 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:49.426832 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-q656v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:50:49.426872 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:49.426906 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podUID="52b663c4-e4ce-433f-80b6-52fe26e9f976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:50:51.472301 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:51.471769 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-q656v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:51.472301 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:51.471852 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" podUID="52b663c4-e4ce-433f-80b6-52fe26e9f976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:51.877302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:51.877211 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:51.877554 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:51.877513 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:51.878714 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:51.878693 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-q656v" Oct 30 18:50:52.176301 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:52.176246 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9"] Oct 30 18:50:53.036792 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:53.036749 5959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:53.037327 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:53.036820 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-bkpg7" podUID="bf0d5ca3-7ba8-481c-b29d-702244848bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:57.782103 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:50:57.782058 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:50:59.546450 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.546412 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs"] Oct 30 18:50:59.552866 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.552836 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.563028 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.562767 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs"] Oct 30 18:50:59.564639 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.564063 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"distinct-w2-39bde3-kserve-self-signed-certs\"" Oct 30 18:50:59.567487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.567459 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tmp-dir\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.567649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.567595 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-home\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.567721 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.567670 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tls-certs\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.567721 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.567705 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt499\" (UniqueName: \"kubernetes.io/projected/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-kube-api-access-zt499\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.567888 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.567846 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-dshm\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.567952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.567918 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-model-cache\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.669521 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.669488 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-home\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.669521 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.669529 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tls-certs\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.669808 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.669558 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt499\" (UniqueName: \"kubernetes.io/projected/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-kube-api-access-zt499\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.669808 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.669643 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-dshm\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.669808 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.669692 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-model-cache\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.669972 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.669834 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tmp-dir\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.670211 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.670188 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tmp-dir\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.670382 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.670352 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-home\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.670767 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.670743 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-model-cache\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.672563 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.672538 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tls-certs\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.672696 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.672660 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-dshm\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.678189 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.678164 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt499\" (UniqueName: \"kubernetes.io/projected/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-kube-api-access-zt499\") pod \"distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:50:59.870371 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:50:59.870288 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:51:00.149837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:00.149812 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs"] Oct 30 18:51:00.152478 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:51:00.152446 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d9ce7f1_3bff_4808_ab73_d1281dd72f98.slice/crio-45e1d4cc0d9eaff352e6e215f75d61ecfaaeb09af941ff20dbc74fa55d2f5397 WatchSource:0}: Error finding container 45e1d4cc0d9eaff352e6e215f75d61ecfaaeb09af941ff20dbc74fa55d2f5397: Status 404 returned error can't find the container with id 45e1d4cc0d9eaff352e6e215f75d61ecfaaeb09af941ff20dbc74fa55d2f5397 Oct 30 18:51:01.130042 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:01.130006 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" event={"ID":"3d9ce7f1-3bff-4808-ab73-d1281dd72f98","Type":"ContainerStarted","Data":"5f8abfb09196767d59a4d43441ea2f96f56d5c66b241f3cd8e0493fb395e89b2"} Oct 30 18:51:01.130447 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:01.130052 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" event={"ID":"3d9ce7f1-3bff-4808-ab73-d1281dd72f98","Type":"ContainerStarted","Data":"45e1d4cc0d9eaff352e6e215f75d61ecfaaeb09af941ff20dbc74fa55d2f5397"} Oct 30 18:51:01.148064 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:01.148011 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podStartSLOduration=2.147997839 podStartE2EDuration="2.147997839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:01.145975722 +0000 UTC m=+841.474387102" watchObservedRunningTime="2026-10-30 18:51:01.147997839 +0000 UTC m=+841.476409224" Oct 30 18:51:01.870805 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:01.870779 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:51:02.626482 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:02.626452 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:51:07.306947 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.306909 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" podUID="6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" containerName="main" containerID="cri-o://810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b" gracePeriod=45 Oct 30 18:51:07.575357 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.575275 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:51:07.581955 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.581932 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-kube-api-access-6md9n\" (UniqueName: \"kubernetes.io/projected/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-kube-api-access-6md9n\") pod \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " Oct 30 18:51:07.582083 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.581989 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-dshm\") pod \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " Oct 30 18:51:07.582083 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582045 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-home\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-home\") pod \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " Oct 30 18:51:07.582194 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582165 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-model-cache\") pod \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " Oct 30 18:51:07.582247 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582215 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tls-certs\") pod \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " Oct 30 18:51:07.582298 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582266 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tmp-dir\") pod \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\" (UID: \"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e\") " Oct 30 18:51:07.582371 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582349 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-home" pod "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" (UID: "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:51:07.582427 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582399 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-model-cache" pod "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" (UID: "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:51:07.582581 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582558 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tmp-dir" pod "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" (UID: "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:51:07.582799 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582730 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:51:07.582799 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582751 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:51:07.582799 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.582765 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:51:07.584338 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.584314 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-dshm" pod "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" (UID: "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:51:07.584450 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.584406 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tls-certs" pod "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" (UID: "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:51:07.584489 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.584462 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-kube-api-access-6md9n" pod "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" (UID: "6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e"). InnerVolumeSpecName "kube-api-access-6md9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:07.683846 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.683812 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:51:07.683846 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.683838 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6md9n\" (UniqueName: \"kubernetes.io/projected/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-kube-api-access-6md9n\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:51:07.683846 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.683848 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:51:07.721971 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.721940 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p"] Oct 30 18:51:07.722424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.722409 5959 cpu_manager.go:414] "containerMap: removing container" podUID="6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" containerName="main" Oct 30 18:51:07.722424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.722423 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" containerName="main" Oct 30 18:51:07.722568 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.722527 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" containerName="main" Oct 30 18:51:07.726148 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.726121 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.737826 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.737790 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p"] Oct 30 18:51:07.784648 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.784614 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.784794 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.784709 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn95\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-kube-api-access-6sn95\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.784794 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.784771 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.784865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.784800 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.784865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.784823 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.784928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.784884 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.784960 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.784949 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.785010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.784995 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.785071 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.785058 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886369 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886344 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886526 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886384 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886526 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886437 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886526 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886475 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn95\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-kube-api-access-6sn95\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886526 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886514 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886554 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886587 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886630 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.886749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.886682 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.887043 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.887016 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.887043 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.887030 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.887172 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.887117 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.887224 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.887163 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.887389 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.887370 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.889137 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.889114 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.889231 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.889221 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.894425 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.894408 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:07.894658 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:07.894639 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn95\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-kube-api-access-6sn95\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:08.045511 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.045475 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:08.194230 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.194145 5959 generic.go:356] "Generic (PLEG): container finished" podID="6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" containerID="810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b" exitCode=0 Oct 30 18:51:08.194230 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.194201 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" Oct 30 18:51:08.194230 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.194214 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" event={"ID":"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e","Type":"ContainerDied","Data":"810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b"} Oct 30 18:51:08.194477 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.194246 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9" event={"ID":"6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e","Type":"ContainerDied","Data":"ec256363960dc580e721ae61a4af8eb7999beea6e4691ef615a7185da72c8aaf"} Oct 30 18:51:08.194477 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.194262 5959 scope.go:122] "RemoveContainer" containerID="810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b" Oct 30 18:51:08.203817 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.203800 5959 scope.go:122] "RemoveContainer" containerID="810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b" Oct 30 18:51:08.204071 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:51:08.204051 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b\": container with ID starting with 810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b not found: ID does not exist" containerID="810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b" Oct 30 18:51:08.204120 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.204078 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b"} err="failed to get container status \"810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b\": rpc error: code = NotFound desc = could not find container \"810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b\": container with ID starting with 810fc8da5df7eb93a2e2456fdd4ed4249db1594290ee9dbcc84f22885270f14b not found: ID does not exist" Oct 30 18:51:08.210923 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.210903 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9"] Oct 30 18:51:08.215309 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.215268 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-a-8eedd2a0-kserve-75d8fc655b-9ckc9"] Oct 30 18:51:08.327295 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.327233 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p"] Oct 30 18:51:08.331143 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:51:08.331112 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea59e1b2_c9a5_4322_83f5_e8c1d9777880.slice/crio-80111b7ca3112502aa9278e64273ad1ac841d6a226f63763d528a7385c99e5a6 WatchSource:0}: Error finding container 80111b7ca3112502aa9278e64273ad1ac841d6a226f63763d528a7385c99e5a6: Status 404 returned error can't find the container with id 80111b7ca3112502aa9278e64273ad1ac841d6a226f63763d528a7385c99e5a6 Oct 30 18:51:08.335562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.335511 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:08.335704 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.335623 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:08.335704 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:08.335667 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:09.231911 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:09.231875 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" event={"ID":"ea59e1b2-c9a5-4322-83f5-e8c1d9777880","Type":"ContainerStarted","Data":"0f22ea7bc924991a9f629da1b08552e22ef0b2f9c9818e4948c29b89d5f41d62"} Oct 30 18:51:09.232194 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:09.232160 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" event={"ID":"ea59e1b2-c9a5-4322-83f5-e8c1d9777880","Type":"ContainerStarted","Data":"80111b7ca3112502aa9278e64273ad1ac841d6a226f63763d528a7385c99e5a6"} Oct 30 18:51:09.255767 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:09.255724 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podStartSLOduration=2.255706716 podStartE2EDuration="2.255706716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:09.253961528 +0000 UTC m=+849.582372913" watchObservedRunningTime="2026-10-30 18:51:09.255706716 +0000 UTC m=+849.584118102" Oct 30 18:51:09.820630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:09.819547 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e" path="/var/lib/kubelet/pods/6bd4aa33-fe70-4078-9fdc-4be2c2dd9c4e/volumes" Oct 30 18:51:09.870786 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:09.870757 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:51:09.883532 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:09.883510 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:51:10.045657 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:10.045573 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:10.047390 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:10.047359 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:51:10.047563 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:10.047420 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 18:51:10.249940 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:10.249900 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:51:11.046228 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:11.046194 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:51:11.046693 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:11.046247 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 18:51:12.045940 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:12.045908 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:51:12.046147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:12.045979 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 18:51:13.046568 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:13.046534 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:51:13.047025 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:13.046599 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 18:51:13.990485 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:13.990448 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk"] Oct 30 18:51:13.994834 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:13.994809 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.027794 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.015319 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk"] Oct 30 18:51:14.050130 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.050090 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:51:14.050658 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.050154 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 18:51:14.077929 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.077895 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.078177 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.078153 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.078253 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.078197 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.078303 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.078271 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.078361 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.078320 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.078411 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.078360 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.078457 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.078419 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.078507 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.078491 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmq7v\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-kube-api-access-cmq7v\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.078563 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.078528 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.180824 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.180787 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.180997 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.180841 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.180997 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.180882 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.180997 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.180928 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmq7v\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-kube-api-access-cmq7v\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.180997 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.180949 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.180997 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.180976 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.181236 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.181042 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.181236 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.181072 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.181236 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.181146 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.181366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.181236 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.181366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.181290 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.181518 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.181494 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.181725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.181697 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.182080 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.181777 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.183532 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.183506 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.183727 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.183704 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.191161 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.191139 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.191532 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.191422 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmq7v\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-kube-api-access-cmq7v\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.352077 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.351986 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:14.660474 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.660447 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk"] Oct 30 18:51:14.662914 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:51:14.662883 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55815db4_66e6_42a3_aa1c_9e49fb47bb06.slice/crio-4e1ea873813b4d254509ae5e054c99f80f1aeabb14b1fb5f6f950aab2bb20614 WatchSource:0}: Error finding container 4e1ea873813b4d254509ae5e054c99f80f1aeabb14b1fb5f6f950aab2bb20614: Status 404 returned error can't find the container with id 4e1ea873813b4d254509ae5e054c99f80f1aeabb14b1fb5f6f950aab2bb20614 Oct 30 18:51:14.665701 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.665668 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:14.665797 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.665736 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:14.665797 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.665762 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:14.703224 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.703189 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff"] Oct 30 18:51:14.710740 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.710712 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.718843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.715726 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"distinct-2-w2-39bde3-kserve-self-signed-certs\"" Oct 30 18:51:14.718843 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.715787 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff"] Oct 30 18:51:14.787892 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.787857 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-home\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.788062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.787943 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-tmp-dir\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.788062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.787980 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b41146-9abf-4010-8c86-c473bbf18e19-tls-certs\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.788062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.788029 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-model-cache\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.788182 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.788146 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh9cg\" (UniqueName: \"kubernetes.io/projected/29b41146-9abf-4010-8c86-c473bbf18e19-kube-api-access-sh9cg\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.788223 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.788193 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-dshm\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.891629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.889709 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh9cg\" (UniqueName: \"kubernetes.io/projected/29b41146-9abf-4010-8c86-c473bbf18e19-kube-api-access-sh9cg\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.891629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.889795 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-dshm\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.891629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.889912 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-home\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.891629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.889974 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-tmp-dir\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.891629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.890022 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b41146-9abf-4010-8c86-c473bbf18e19-tls-certs\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.891629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.890071 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-model-cache\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.891629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.890650 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-model-cache\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.892083 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.892012 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-home\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.897625 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.892689 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-tmp-dir\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.897625 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.895180 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b41146-9abf-4010-8c86-c473bbf18e19-tls-certs\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.897625 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.895329 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-dshm\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:14.900145 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:14.900098 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh9cg\" (UniqueName: \"kubernetes.io/projected/29b41146-9abf-4010-8c86-c473bbf18e19-kube-api-access-sh9cg\") pod \"distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:15.035636 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.035530 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:15.046397 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.046366 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 18:51:15.046719 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.046686 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 18:51:15.274028 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.273986 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" event={"ID":"55815db4-66e6-42a3-aa1c-9e49fb47bb06","Type":"ContainerStarted","Data":"95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981"} Oct 30 18:51:15.274460 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.274101 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" event={"ID":"55815db4-66e6-42a3-aa1c-9e49fb47bb06","Type":"ContainerStarted","Data":"4e1ea873813b4d254509ae5e054c99f80f1aeabb14b1fb5f6f950aab2bb20614"} Oct 30 18:51:15.294629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.294471 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podStartSLOduration=2.294455289 podStartE2EDuration="2.294455289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:15.294072397 +0000 UTC m=+855.622483794" watchObservedRunningTime="2026-10-30 18:51:15.294455289 +0000 UTC m=+855.622866675" Oct 30 18:51:15.352570 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.352530 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:15.354094 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.354063 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:51:15.354243 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.354118 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:51:15.369141 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:15.369001 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff"] Oct 30 18:51:15.373034 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:51:15.371944 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29b41146_9abf_4010_8c86_c473bbf18e19.slice/crio-63832821f562df755bc41b7153f95193b84a973a97e0951c83da29cf4b271805 WatchSource:0}: Error finding container 63832821f562df755bc41b7153f95193b84a973a97e0951c83da29cf4b271805: Status 404 returned error can't find the container with id 63832821f562df755bc41b7153f95193b84a973a97e0951c83da29cf4b271805 Oct 30 18:51:16.178412 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:16.178375 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:16.179351 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:16.179323 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:16.180478 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:16.180453 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 18:51:16.285005 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:16.284971 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" event={"ID":"29b41146-9abf-4010-8c86-c473bbf18e19","Type":"ContainerStarted","Data":"c8258316140d50ae03e54c5f30616be0e47b7a4c154272344fb91aa95b963dd7"} Oct 30 18:51:16.285442 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:16.285016 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" event={"ID":"29b41146-9abf-4010-8c86-c473bbf18e19","Type":"ContainerStarted","Data":"63832821f562df755bc41b7153f95193b84a973a97e0951c83da29cf4b271805"} Oct 30 18:51:16.352498 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:16.352470 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:51:16.352680 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:16.352520 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:51:17.036365 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:17.036333 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:17.353408 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:17.353323 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:51:17.353408 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:17.353386 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:51:18.352550 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:18.352516 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:51:18.352733 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:18.352580 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:51:19.353062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:19.353027 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:51:19.353454 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:19.353086 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:51:20.353314 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:20.353280 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:51:20.353799 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:20.353337 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:51:21.352428 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:21.352378 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:51:21.352638 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:21.352478 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:51:21.645940 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:21.645877 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podStartSLOduration=7.645855845 podStartE2EDuration="7.645855845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:16.305574952 +0000 UTC m=+856.633986336" watchObservedRunningTime="2026-10-30 18:51:21.645855845 +0000 UTC m=+861.974267230" Oct 30 18:51:21.647592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:21.647566 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:51:22.356686 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.356661 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:22.356945 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.356925 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:22.357540 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.357522 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 18:51:22.773161 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.773131 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4"] Oct 30 18:51:22.814200 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.814168 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4"] Oct 30 18:51:22.857756 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.822254 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.953366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953322 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.953560 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953399 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.953560 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953460 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5lts\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-kube-api-access-s5lts\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.953560 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953499 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.953560 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953546 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.953846 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953709 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.953846 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953773 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.953846 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953818 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:22.954004 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:22.953908 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.054849 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.054917 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lts\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-kube-api-access-s5lts\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.054955 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.055002 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.055067 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.055119 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.055161 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.055256 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.055689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.055338 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.087629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.080789 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.107823 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.088368 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.108010 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.089108 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.108105 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.089691 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.108219 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.090138 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.108319 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.090362 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.108421 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.099326 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.134630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.132481 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.138477 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.138451 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5lts\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-kube-api-access-s5lts\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:23.216824 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:23.216793 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:24.007852 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:24.007774 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4"] Oct 30 18:51:24.047970 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:51:24.047941 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbb74a6_5c3e_4bf5_ae5b_b8fbcb708422.slice/crio-44ca3d14a5ba654dbf5d63e773f4ef321bdaee9ce89de84562e31cdcb4fa0987 WatchSource:0}: Error finding container 44ca3d14a5ba654dbf5d63e773f4ef321bdaee9ce89de84562e31cdcb4fa0987: Status 404 returned error can't find the container with id 44ca3d14a5ba654dbf5d63e773f4ef321bdaee9ce89de84562e31cdcb4fa0987 Oct 30 18:51:24.058226 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:24.057428 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:24.058226 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:24.057521 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:24.058226 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:24.057580 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:24.532577 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:24.532486 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" event={"ID":"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422","Type":"ContainerStarted","Data":"7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb"} Oct 30 18:51:24.532982 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:24.532929 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" event={"ID":"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422","Type":"ContainerStarted","Data":"44ca3d14a5ba654dbf5d63e773f4ef321bdaee9ce89de84562e31cdcb4fa0987"} Oct 30 18:51:24.557688 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:24.557635 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podStartSLOduration=2.557616156 podStartE2EDuration="2.557616156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:24.556711061 +0000 UTC m=+864.885122463" watchObservedRunningTime="2026-10-30 18:51:24.557616156 +0000 UTC m=+864.886027531" Oct 30 18:51:25.036620 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:25.036571 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:25.051132 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:25.051100 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:25.219856 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:25.219822 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:25.221816 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:25.221785 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:51:25.222062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:25.222035 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:51:25.547225 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:25.547192 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:26.218007 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:26.217969 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:51:26.218626 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:26.218572 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:51:27.224631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.218119 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:51:27.224631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.218204 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:51:27.224631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.221300 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9"] Oct 30 18:51:27.233837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.233809 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9"] Oct 30 18:51:27.233994 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.233944 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.353854 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.353792 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.354023 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.353961 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.354023 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.353988 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.354133 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.354019 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.354190 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.354168 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.354243 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.354227 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.354314 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.354294 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c108933-c3fb-4fbd-97da-982149fedf1f-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.354349 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.354337 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.354392 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.354354 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfp44\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-kube-api-access-mfp44\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.455695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.455658 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c108933-c3fb-4fbd-97da-982149fedf1f-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.455695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.455701 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.455917 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.455716 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfp44\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-kube-api-access-mfp44\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.455917 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.455814 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.455917 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.455893 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456079 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.455916 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456079 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.455944 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456079 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.455994 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456079 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.456051 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456295 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.456268 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456371 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.456297 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456437 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.456378 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456437 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.456421 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c108933-c3fb-4fbd-97da-982149fedf1f-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.456529 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.456454 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.458240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.458216 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.458340 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.458255 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.463581 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.463555 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.463707 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.463691 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfp44\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-kube-api-access-mfp44\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.552948 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.552870 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:27.780377 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:51:27.780343 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:51:27.867655 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.867589 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9"] Oct 30 18:51:27.869942 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:51:27.869914 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c108933_c3fb_4fbd_97da_982149fedf1f.slice/crio-56dfeb9f3853074145606ee4de0f2aa4e9c0891268bbf8ba41d98c87f6f889ac WatchSource:0}: Error finding container 56dfeb9f3853074145606ee4de0f2aa4e9c0891268bbf8ba41d98c87f6f889ac: Status 404 returned error can't find the container with id 56dfeb9f3853074145606ee4de0f2aa4e9c0891268bbf8ba41d98c87f6f889ac Oct 30 18:51:27.872439 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.872412 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:27.872511 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.872475 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:27.872543 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:27.872518 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:28.222166 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:28.219440 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:51:28.222166 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:28.219497 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:51:28.560258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:28.560171 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" event={"ID":"8c108933-c3fb-4fbd-97da-982149fedf1f","Type":"ContainerStarted","Data":"7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581"} Oct 30 18:51:28.560258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:28.560229 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" event={"ID":"8c108933-c3fb-4fbd-97da-982149fedf1f","Type":"ContainerStarted","Data":"56dfeb9f3853074145606ee4de0f2aa4e9c0891268bbf8ba41d98c87f6f889ac"} Oct 30 18:51:28.580846 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:28.580799 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podStartSLOduration=1.580787429 podStartE2EDuration="1.580787429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:28.579752503 +0000 UTC m=+868.908163900" watchObservedRunningTime="2026-10-30 18:51:28.580787429 +0000 UTC m=+868.909198816" Oct 30 18:51:29.218542 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:29.218049 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:51:29.218542 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:29.218113 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:51:29.553809 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:29.553714 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:29.555029 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:29.555005 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:51:29.555105 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:29.555064 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:51:30.226532 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:30.226499 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:51:30.227187 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:30.227159 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:51:30.553445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:30.553350 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:51:30.553445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:30.553414 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:51:31.218028 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:31.217988 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:51:31.218299 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:31.218272 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:51:31.553251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:31.553151 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:51:31.553251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:31.553221 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:51:32.225322 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:32.225290 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:32.225590 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:32.225570 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:32.226449 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:32.226429 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 18:51:32.553499 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:32.553405 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:51:32.553499 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:32.553458 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:51:33.553583 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:33.553549 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:51:33.554097 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:33.553641 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:51:34.553790 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:34.553760 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:51:34.554274 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:34.553818 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:51:35.558353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:35.558318 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:35.558880 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:35.558815 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:35.559418 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:35.559398 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 18:51:41.252629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:41.247808 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 18:51:41.336631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:41.332465 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:41.347638 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:41.347382 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:41.401251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:41.401202 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 18:51:41.861632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:41.859705 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:42.325632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.322888 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5"] Oct 30 18:51:42.587789 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.580846 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5"] Oct 30 18:51:42.594675 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.583807 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.672401 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.690827 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.690916 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.691004 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.691048 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.691088 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a468ed9d-e51f-422b-927d-abce4d0c7a36-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.691127 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.691182 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.691218 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmcg\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-kube-api-access-hfmcg\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.700632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.695492 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t"] Oct 30 18:51:42.741220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.741187 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.784443 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.784413 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9de872\"/\"distinct-w0-9de872-kserve-self-signed-certs\"" Oct 30 18:51:42.843704 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.827803 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t"] Oct 30 18:51:42.846968 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.846761 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.847265 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.847225 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhdk\" (UniqueName: \"kubernetes.io/projected/e300afa7-3bad-4988-a244-2c0dfb446993-kube-api-access-tmhdk\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.847441 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.847423 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-model-cache\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.847567 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.847552 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.847722 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.847705 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.847849 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.847829 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e300afa7-3bad-4988-a244-2c0dfb446993-tls-certs\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.847980 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.847965 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a468ed9d-e51f-422b-927d-abce4d0c7a36-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.848101 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.848085 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.848247 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.848228 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.848369 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.848353 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmcg\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-kube-api-access-hfmcg\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.848526 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.848509 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-dshm\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.848739 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.848720 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.848906 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.848888 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.849041 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.849025 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-home\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.849157 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.849143 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-tmp-dir\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.849840 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.849803 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.850265 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.850244 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.850754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.850733 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.851591 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.851569 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a468ed9d-e51f-422b-927d-abce4d0c7a36-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.851998 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.851979 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.985633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.959910 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:42.985633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.960807 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-home\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.985633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.960844 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-tmp-dir\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.985633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.960909 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhdk\" (UniqueName: \"kubernetes.io/projected/e300afa7-3bad-4988-a244-2c0dfb446993-kube-api-access-tmhdk\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.985633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.960963 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-model-cache\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.985633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.961009 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e300afa7-3bad-4988-a244-2c0dfb446993-tls-certs\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.985633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.961117 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-dshm\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:42.989332 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.989302 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-home\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:43.023030 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.989752 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-tmp-dir\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:43.023230 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:42.989888 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-model-cache\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:43.023381 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.012887 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:43.023489 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.013476 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmcg\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-kube-api-access-hfmcg\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:43.031660 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.031584 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-dshm\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:43.038801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.038714 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhdk\" (UniqueName: \"kubernetes.io/projected/e300afa7-3bad-4988-a244-2c0dfb446993-kube-api-access-tmhdk\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:43.058637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.050421 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:43.058637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.057577 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e300afa7-3bad-4988-a244-2c0dfb446993-tls-certs\") pod \"distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:43.218882 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.218582 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:43.291499 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.291458 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:43.632360 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.630903 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5"] Oct 30 18:51:43.636101 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:51:43.636072 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda468ed9d_e51f_422b_927d_abce4d0c7a36.slice/crio-2487599d3056266032286e94239cc5cc1d5b8715d2cd70f3d4a3b69ab550ffa6 WatchSource:0}: Error finding container 2487599d3056266032286e94239cc5cc1d5b8715d2cd70f3d4a3b69ab550ffa6: Status 404 returned error can't find the container with id 2487599d3056266032286e94239cc5cc1d5b8715d2cd70f3d4a3b69ab550ffa6 Oct 30 18:51:43.642642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.640270 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:43.642642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.640367 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:43.642642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.640412 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:51:43.834848 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:43.832367 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t"] Oct 30 18:51:44.039975 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.039933 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" event={"ID":"e300afa7-3bad-4988-a244-2c0dfb446993","Type":"ContainerStarted","Data":"5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949"} Oct 30 18:51:44.040175 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.039994 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" event={"ID":"e300afa7-3bad-4988-a244-2c0dfb446993","Type":"ContainerStarted","Data":"034e9e003c19cfcb0830804d57351137bc566aee41eeea3d0768695c45ad586e"} Oct 30 18:51:44.044187 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.044151 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" event={"ID":"a468ed9d-e51f-422b-927d-abce4d0c7a36","Type":"ContainerStarted","Data":"9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105"} Oct 30 18:51:44.045205 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.045179 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" event={"ID":"a468ed9d-e51f-422b-927d-abce4d0c7a36","Type":"ContainerStarted","Data":"2487599d3056266032286e94239cc5cc1d5b8715d2cd70f3d4a3b69ab550ffa6"} Oct 30 18:51:44.063239 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.061474 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podStartSLOduration=2.061452962 podStartE2EDuration="2.061452962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:44.060271795 +0000 UTC m=+884.388683187" watchObservedRunningTime="2026-10-30 18:51:44.061452962 +0000 UTC m=+884.389864349" Oct 30 18:51:44.090991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.090923 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podStartSLOduration=2.090890782 podStartE2EDuration="2.090890782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:44.089970796 +0000 UTC m=+884.418382199" watchObservedRunningTime="2026-10-30 18:51:44.090890782 +0000 UTC m=+884.419302169" Oct 30 18:51:44.219438 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.219399 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:44.221034 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.221007 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:51:44.221168 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.221056 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 18:51:44.292760 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:44.292725 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:45.220531 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:45.220078 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:51:45.220531 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:45.220147 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 18:51:46.221032 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:46.220997 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:51:46.221545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:46.221057 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 18:51:47.220634 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:47.220036 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:51:47.220634 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:47.220101 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 18:51:48.221147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:48.221110 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:51:48.221574 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:48.221178 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 18:51:49.223104 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:49.223057 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:51:49.223657 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:49.223131 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 18:51:50.219422 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:50.219383 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:51:50.219649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:50.219441 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 18:51:51.330658 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:51.330587 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:51.331144 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:51.331123 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:51.332121 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:51.332104 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 18:51:53.292359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:53.292317 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:53.305466 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:53.305444 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:54.116597 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:54.116554 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:55.131293 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:55.131241 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:51:57.657370 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:57.652730 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:58.027081 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:51:58.026979 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:51:58.110116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:58.110056 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:58.227764 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:58.227717 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:58.246630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:58.244126 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:58.246630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:58.244471 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 18:51:58.257635 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:58.253519 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 18:51:58.365700 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:58.365565 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:58.596209 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:58.596163 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:58.798823 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:58.798767 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:59.122169 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.122065 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h"] Oct 30 18:51:59.137874 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.137832 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:59.244637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.233741 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:59.246785 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.246741 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:59.266901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.251819 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 18:51:59.266901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.255705 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:59.321634 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.315871 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h"] Oct 30 18:51:59.325638 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.321928 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:59.329633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.325910 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:51:59.329885 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.329866 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:51:59.360632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.333687 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:51:59.386374 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.386348 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:51:59.386664 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.386646 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:51:59.387633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.318694 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.499313 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.499276 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2sxk\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-kube-api-access-h2sxk\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.548595 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.548556 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.548929 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.548905 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.663924 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.663829 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.663924 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.663890 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.664215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.663949 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.664215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.663989 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.664215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.664097 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.664215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.664128 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.664418 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.596689 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:51:59.680847 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.680819 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.787533 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2sxk\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-kube-api-access-h2sxk\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.787622 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.787684 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.787819 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.787895 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.787949 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.788007 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.788117 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.788172 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.788728 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:51:59.816282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.789315 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.058817 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.894981 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.058817 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:51:59.895697 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.171705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.105449 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:52:00.171705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.145065 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:52:00.171705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.145577 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:52:00.171705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.163134 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.234265 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.217963 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.234265 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.218570 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2sxk\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-kube-api-access-h2sxk\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.234265 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.219079 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.280386 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.278772 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:00.340004 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.339878 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.494871 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.482422 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:00.565079 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.564935 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:00.691709 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.691676 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:52:00.710490 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.694635 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:52:00.733020 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:00.724554 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:52:01.104992 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:01.104858 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h"] Oct 30 18:52:01.107840 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:52:01.107810 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea10d0b_617b_4aa0_a73d_5d22a0e7153a.slice/crio-e88310e014b8fec2ce23bd1d5e8615d5a7367c3593de0492440f7acab71c880d WatchSource:0}: Error finding container e88310e014b8fec2ce23bd1d5e8615d5a7367c3593de0492440f7acab71c880d: Status 404 returned error can't find the container with id e88310e014b8fec2ce23bd1d5e8615d5a7367c3593de0492440f7acab71c880d Oct 30 18:52:01.110556 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:01.110522 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:01.110695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:01.110672 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:01.110769 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:01.110724 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:01.315270 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:01.315218 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" event={"ID":"aea10d0b-617b-4aa0-a73d-5d22a0e7153a","Type":"ContainerStarted","Data":"0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360"} Oct 30 18:52:01.315463 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:01.315278 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" event={"ID":"aea10d0b-617b-4aa0-a73d-5d22a0e7153a","Type":"ContainerStarted","Data":"e88310e014b8fec2ce23bd1d5e8615d5a7367c3593de0492440f7acab71c880d"} Oct 30 18:52:01.338896 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:01.338847 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" podStartSLOduration=2.338830869 podStartE2EDuration="2.338830869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:01.336824706 +0000 UTC m=+901.665236091" watchObservedRunningTime="2026-10-30 18:52:01.338830869 +0000 UTC m=+901.667242255" Oct 30 18:52:01.565713 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:01.565686 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:02.568789 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:02.568754 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:52:02.569347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:02.568818 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:52:03.566653 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:03.566598 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:52:03.566835 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:03.566670 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:52:04.566280 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:04.566236 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:52:04.566767 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:04.566301 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:52:05.566485 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:05.566453 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:52:05.566922 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:05.566508 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:52:06.566299 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:06.566267 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:52:06.566515 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:06.566321 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:52:07.566443 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:07.566403 5959 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:52:07.566651 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:07.566470 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:52:08.729781 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:08.729752 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:08.730741 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:08.730717 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:08.732768 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:08.732748 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 18:52:09.040812 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.040723 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj"] Oct 30 18:52:09.052747 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.052720 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.055150 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.055122 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj"] Oct 30 18:52:09.069072 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.069013 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"distinct-w1-85cb41-kserve-self-signed-certs\"" Oct 30 18:52:09.091139 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.091101 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-home\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.091299 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.091157 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fms\" (UniqueName: \"kubernetes.io/projected/96ba3eb5-1b12-4e2b-9fec-9842167e514a-kube-api-access-58fms\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.091299 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.091280 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-dshm\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.091421 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.091399 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tmp-dir\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.091553 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.091536 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tls-certs\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.091646 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.091625 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-model-cache\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.199558 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.199516 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tmp-dir\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.199815 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.199591 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tls-certs\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.199815 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.199675 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-model-cache\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.199815 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.199798 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-home\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.199992 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.199842 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fms\" (UniqueName: \"kubernetes.io/projected/96ba3eb5-1b12-4e2b-9fec-9842167e514a-kube-api-access-58fms\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.199992 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.199873 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-dshm\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.202019 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.201987 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-model-cache\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.202577 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.202550 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tmp-dir\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.203395 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.203371 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-home\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.212876 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.212820 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fms\" (UniqueName: \"kubernetes.io/projected/96ba3eb5-1b12-4e2b-9fec-9842167e514a-kube-api-access-58fms\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.234145 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.234116 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tls-certs\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.235427 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.234633 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-dshm\") pod \"distinct-w1-85cb41-kserve-674dc75c77-tfcjj\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.381829 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.381748 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:09.663758 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:09.663729 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj"] Oct 30 18:52:09.666479 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:52:09.666450 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96ba3eb5_1b12_4e2b_9fec_9842167e514a.slice/crio-3ead26c670601e0f115a80c32527b8467e3db81dbb8c1a5c741d46dea70be6d6 WatchSource:0}: Error finding container 3ead26c670601e0f115a80c32527b8467e3db81dbb8c1a5c741d46dea70be6d6: Status 404 returned error can't find the container with id 3ead26c670601e0f115a80c32527b8467e3db81dbb8c1a5c741d46dea70be6d6 Oct 30 18:52:10.381933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:10.381848 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" event={"ID":"96ba3eb5-1b12-4e2b-9fec-9842167e514a","Type":"ContainerStarted","Data":"ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8"} Oct 30 18:52:10.381933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:10.381894 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" event={"ID":"96ba3eb5-1b12-4e2b-9fec-9842167e514a","Type":"ContainerStarted","Data":"3ead26c670601e0f115a80c32527b8467e3db81dbb8c1a5c741d46dea70be6d6"} Oct 30 18:52:10.399963 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:10.399915 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podStartSLOduration=1.399901936 podStartE2EDuration="1.399901936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:10.398216754 +0000 UTC m=+910.726628141" watchObservedRunningTime="2026-10-30 18:52:10.399901936 +0000 UTC m=+910.728313320" Oct 30 18:52:11.382352 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:11.382319 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:13.793635 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:13.792685 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 18:52:14.080443 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:14.075689 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:14.117686 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:14.111329 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:14.142453 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:14.137720 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:14.296632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:14.293711 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:14.307275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:14.307138 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:14.799630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:14.798704 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:14.815635 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:14.800623 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:52:15.126691 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.097424 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:15.169630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.127689 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:52:15.169630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.165804 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:52:15.169630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.165996 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:52:15.192507 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.138835 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:15.193572 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.193521 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:52:15.332632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.327696 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 18:52:15.332632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.328471 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:52:15.544747 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.544712 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:52:15.572629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.572429 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:52:15.778836 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.776810 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:15.803681 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.799738 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:15.876133 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.876102 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:52:15.913574 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.913532 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:52:15.942406 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:15.941480 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:52:19.382856 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:19.382825 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:19.395783 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:19.395762 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:19.672221 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:19.672190 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:23.489228 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:23.489197 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:52:27.782957 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:52:27.782913 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:52:28.525394 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.525360 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc"] Oct 30 18:52:28.530753 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.530722 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.537357 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.537323 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9de872-openshift-default-dockercfg-9bf8l\"" Oct 30 18:52:28.544599 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.544540 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc"] Oct 30 18:52:28.720654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.720584 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.720821 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.720702 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhh2\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-kube-api-access-5zhh2\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.720821 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.720780 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.720916 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.720897 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.720959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.720946 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.720993 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.720969 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.721087 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.721076 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.721127 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.721115 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d5d58c-3291-49e5-9e98-e2989a67316a-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.721170 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.721159 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822166 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822227 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822256 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822355 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822378 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d5d58c-3291-49e5-9e98-e2989a67316a-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822483 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822553 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822595 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhh2\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-kube-api-access-5zhh2\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822678 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.822806 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822707 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.823173 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.822846 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.823173 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.823074 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d5d58c-3291-49e5-9e98-e2989a67316a-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.823173 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.823098 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.823173 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.823116 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.824739 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.824717 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.825089 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.825070 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.831106 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.831075 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.831293 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.831273 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhh2\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-kube-api-access-5zhh2\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:28.853237 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:28.853206 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:29.143581 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:29.143555 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc"] Oct 30 18:52:29.146242 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:52:29.146215 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d5d58c_3291_49e5_9e98_e2989a67316a.slice/crio-33f5d2fcd29ea8b0c2d245b29fb4d3ff5cc50f00f214e121a96c3a229180b8ea WatchSource:0}: Error finding container 33f5d2fcd29ea8b0c2d245b29fb4d3ff5cc50f00f214e121a96c3a229180b8ea: Status 404 returned error can't find the container with id 33f5d2fcd29ea8b0c2d245b29fb4d3ff5cc50f00f214e121a96c3a229180b8ea Oct 30 18:52:29.149250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:29.149222 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:29.149350 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:29.149285 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:29.149350 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:29.149313 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:29.748330 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:29.748290 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" event={"ID":"45d5d58c-3291-49e5-9e98-e2989a67316a","Type":"ContainerStarted","Data":"417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2"} Oct 30 18:52:29.748518 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:29.748358 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" event={"ID":"45d5d58c-3291-49e5-9e98-e2989a67316a","Type":"ContainerStarted","Data":"33f5d2fcd29ea8b0c2d245b29fb4d3ff5cc50f00f214e121a96c3a229180b8ea"} Oct 30 18:52:29.768952 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:29.768901 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podStartSLOduration=1.7688869569999999 podStartE2EDuration="1.768886957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:29.767354734 +0000 UTC m=+930.095766120" watchObservedRunningTime="2026-10-30 18:52:29.768886957 +0000 UTC m=+930.097298341" Oct 30 18:52:29.854218 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:29.854177 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:30.855914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:30.855880 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:52:30.856303 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:30.855943 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:52:31.854414 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:31.854379 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:52:31.854565 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:31.854438 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:52:32.856633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:32.854647 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:52:32.856633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:32.854694 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:52:33.853800 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:33.853765 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:52:33.854017 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:33.853833 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:52:34.853829 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:34.853732 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:52:34.854322 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:34.853863 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:52:35.854246 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:35.854207 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:52:35.854783 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:35.854285 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:52:36.857964 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:36.857937 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:36.858346 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:36.858291 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:36.859117 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:36.859098 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 18:52:39.944744 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:39.944697 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:40.589703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:40.589657 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 18:52:40.817737 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:40.817690 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:40.886293 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:40.886258 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:40.916727 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:40.916683 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 18:52:40.947538 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:40.947494 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:41.097765 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:41.097716 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:41.460470 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:41.424769 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:41.560248 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:41.560205 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:41.592015 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:41.591970 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:41.592948 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:41.592925 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:52:41.669388 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:41.669339 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 18:52:41.842684 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:41.828475 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:41.983631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:41.969937 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:42.007627 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.003635 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:42.172935 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.172905 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:52:42.173455 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.173437 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:52:42.174481 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.174465 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:52:42.179322 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.179304 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:52:42.561622 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.561505 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:42.572236 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.572205 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:52:42.689378 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.689317 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:52:42.691781 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.691751 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:52:42.692031 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.692016 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:42.774111 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.750046 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:52:42.876830 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.876738 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:52:42.948629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.948198 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:52:42.989994 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:42.989961 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:52:43.013415 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:43.013391 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:52:44.314675 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.314640 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk"] Oct 30 18:52:44.319875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.319856 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.326899 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.326876 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-85cb41-openshift-default-dockercfg-s655r\"" Oct 30 18:52:44.331397 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.331372 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk"] Oct 30 18:52:44.464050 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464017 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.464210 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464066 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.464210 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464088 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.464283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464230 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988rz\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-kube-api-access-988rz\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.464283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464265 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.464381 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464282 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.464381 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464348 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.464483 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464431 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.464483 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.464478 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566048 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.565971 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988rz\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-kube-api-access-988rz\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566048 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566018 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566253 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566046 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566253 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566078 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566253 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566107 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566253 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566153 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566468 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566252 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566468 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566305 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566468 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566335 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566641 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566466 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566641 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566561 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566753 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566719 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566797 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566750 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.566910 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.566891 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.568640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.568595 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.568750 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.568673 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.573377 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.573350 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.573649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.573596 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-988rz\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-kube-api-access-988rz\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.636452 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.636419 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:44.922190 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.922162 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk"] Oct 30 18:52:44.924654 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:52:44.924623 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae6438c_e0eb_4ac2_b04f_6e61c6bb504e.slice/crio-d5ef03712c7b05bbe6da6b94621b54ba3f46b8ed4e140464296126a079dbab25 WatchSource:0}: Error finding container d5ef03712c7b05bbe6da6b94621b54ba3f46b8ed4e140464296126a079dbab25: Status 404 returned error can't find the container with id d5ef03712c7b05bbe6da6b94621b54ba3f46b8ed4e140464296126a079dbab25 Oct 30 18:52:44.927533 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.927501 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:44.927683 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.927560 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:44.927683 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:44.927586 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:45.124331 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:45.124293 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" event={"ID":"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e","Type":"ContainerStarted","Data":"88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b"} Oct 30 18:52:45.124507 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:45.124354 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" event={"ID":"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e","Type":"ContainerStarted","Data":"d5ef03712c7b05bbe6da6b94621b54ba3f46b8ed4e140464296126a079dbab25"} Oct 30 18:52:45.145858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:45.145803 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" podStartSLOduration=1.145783232 podStartE2EDuration="1.145783232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:45.143311309 +0000 UTC m=+945.471722696" watchObservedRunningTime="2026-10-30 18:52:45.145783232 +0000 UTC m=+945.474194617" Oct 30 18:52:45.637252 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:45.637219 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:46.638494 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:46.638461 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:52:46.638918 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:46.638506 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:52:47.639715 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:47.638239 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:52:47.639715 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:47.638315 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:52:48.636800 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:48.636763 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:52:48.636988 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:48.636833 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:52:49.637089 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:49.637056 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:52:49.637649 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:49.637593 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:52:50.637009 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:50.636975 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:52:50.637191 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:50.637042 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:52:51.637160 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:51.637119 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:52:51.637353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:51.637187 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:52:52.700813 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:52.700782 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:52.701266 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:52.701249 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:52.702303 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:52.702285 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 18:52:57.124171 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.124140 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn"] Oct 30 18:52:57.128405 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.128379 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:57.139235 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.139190 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn"] Oct 30 18:52:57.149047 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.149013 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-87293f-dockercfg-zgwpp\"" Oct 30 18:52:57.206755 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.206706 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-87293f\"" Oct 30 18:52:57.248128 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.248092 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daea56e1-2b7e-45dd-8554-727a7546d84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn\" (UID: \"daea56e1-2b7e-45dd-8554-727a7546d84c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:57.248322 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.248239 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxvd\" (UniqueName: \"kubernetes.io/projected/daea56e1-2b7e-45dd-8554-727a7546d84c-kube-api-access-cmxvd\") pod \"payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn\" (UID: \"daea56e1-2b7e-45dd-8554-727a7546d84c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:57.349563 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.349525 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daea56e1-2b7e-45dd-8554-727a7546d84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn\" (UID: \"daea56e1-2b7e-45dd-8554-727a7546d84c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:57.349748 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.349635 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxvd\" (UniqueName: \"kubernetes.io/projected/daea56e1-2b7e-45dd-8554-727a7546d84c-kube-api-access-cmxvd\") pod \"payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn\" (UID: \"daea56e1-2b7e-45dd-8554-727a7546d84c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:57.350255 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.350234 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daea56e1-2b7e-45dd-8554-727a7546d84c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn\" (UID: \"daea56e1-2b7e-45dd-8554-727a7546d84c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:57.359390 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.359365 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxvd\" (UniqueName: \"kubernetes.io/projected/daea56e1-2b7e-45dd-8554-727a7546d84c-kube-api-access-cmxvd\") pod \"payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn\" (UID: \"daea56e1-2b7e-45dd-8554-727a7546d84c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:57.444168 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.444140 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:57.730622 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:57.728204 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn"] Oct 30 18:52:57.785071 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:52:57.785032 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:52:58.218711 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:58.218677 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" event={"ID":"daea56e1-2b7e-45dd-8554-727a7546d84c","Type":"ContainerStarted","Data":"cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc"} Oct 30 18:52:58.219132 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:58.218735 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" event={"ID":"daea56e1-2b7e-45dd-8554-727a7546d84c","Type":"ContainerStarted","Data":"c74de60a4e41ba7ff4a7922916801af6bc96ef73e8a7c6a5b6ecca49897dca5e"} Oct 30 18:52:58.219132 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:58.218872 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:52:58.235757 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:58.235702 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" podStartSLOduration=1.23568438 podStartE2EDuration="1.23568438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:58.235257028 +0000 UTC m=+958.563668407" watchObservedRunningTime="2026-10-30 18:52:58.23568438 +0000 UTC m=+958.564095767" Oct 30 18:52:59.167776 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.167741 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:52:59.333673 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.333640 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg"] Oct 30 18:52:59.338128 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.338106 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.349767 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.349743 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg"] Oct 30 18:52:59.484389 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484309 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpgg4\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-kube-api-access-dpgg4\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.484389 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484361 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.484576 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484435 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.484576 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484462 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.484576 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484483 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.484708 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484635 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.484708 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484671 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.484786 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484757 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.484786 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.484782 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.586487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586394 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.586487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586434 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.586487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586482 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586503 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586586 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpgg4\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-kube-api-access-dpgg4\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586674 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586701 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586725 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586851 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586877 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.586958 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.587106 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.587159 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.587163 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.589085 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.589378 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.594144 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpgg4\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-kube-api-access-dpgg4\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.603858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.594229 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.653427 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.653400 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:52:59.945942 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.945908 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg"] Oct 30 18:52:59.947236 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:52:59.947209 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b59a8b4_a585_47b3_bc59_a48c4cce09e9.slice/crio-6cb992e8058a859423b166c56094d6c9fbaccb36da0c9713070b3e3418cb79fe WatchSource:0}: Error finding container 6cb992e8058a859423b166c56094d6c9fbaccb36da0c9713070b3e3418cb79fe: Status 404 returned error can't find the container with id 6cb992e8058a859423b166c56094d6c9fbaccb36da0c9713070b3e3418cb79fe Oct 30 18:52:59.949978 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.949946 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:59.950082 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.950019 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:52:59.950082 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:52:59.950069 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:00.259689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:00.248559 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" event={"ID":"3b59a8b4-a585-47b3-bc59-a48c4cce09e9","Type":"ContainerStarted","Data":"2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372"} Oct 30 18:53:00.259689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:00.248696 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" event={"ID":"3b59a8b4-a585-47b3-bc59-a48c4cce09e9","Type":"ContainerStarted","Data":"6cb992e8058a859423b166c56094d6c9fbaccb36da0c9713070b3e3418cb79fe"} Oct 30 18:53:00.269087 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:00.269018 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podStartSLOduration=1.268999504 podStartE2EDuration="1.268999504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:00.267677612 +0000 UTC m=+960.596089019" watchObservedRunningTime="2026-10-30 18:53:00.268999504 +0000 UTC m=+960.597410892" Oct 30 18:53:00.668537 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:00.664853 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:53:01.657435 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:01.657398 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:01.657758 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:01.657726 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:01.952825 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:01.952715 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:02.006632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:01.996386 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:02.032631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.018734 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:02.091640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.083743 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:02.180754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.180706 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 18:53:02.412920 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.372736 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:02.470685 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.469749 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" podUID="193f879e-f190-4a8d-8b3b-dcf68accc4ef" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.25:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:02.478850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.478715 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:53:02.478850 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.478781 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:02.679725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.659358 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:02.679725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.659421 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:02.732652 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.724681 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 18:53:02.732652 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.725905 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.025691 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.969874 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.025691 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.992879 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.025691 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:02.993586 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:53:03.025691 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.022836 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.025691 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.024383 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:53:03.079524 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.027454 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.079524 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.027830 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:53:03.091637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.084046 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 18:53:03.091637 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.084706 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:53:03.192633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.189461 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.210635 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.198464 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:53:03.295697 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.295168 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:53:03.295697 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.295234 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.416634 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.415935 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.416813 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.416789 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:53:03.751813 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.750460 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:03.751813 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.750524 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:03.777877 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.776619 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 18:53:03.868985 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.798776 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:03.868985 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.806632 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:53:03.868985 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.806702 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:53:04.021122 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:03.972297 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:04.021122 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.019625 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 18:53:04.024279 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.024248 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:53:04.024485 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.024471 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:53:04.024655 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.024642 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:53:04.090271 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.027210 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:04.090271 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.028752 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:04.329333 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.329233 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Oct 30 18:53:04.482164 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.480882 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:04.689703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.686264 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:04.689703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.686337 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:04.795630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.790711 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:04.800800 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.800756 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:04.835633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.834685 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:04.972596 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.972495 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:04.975372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:04.972694 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:53:05.024803 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.023687 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:05.028710 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.028673 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:05.029582 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.029558 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:05.385713 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.385587 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 18:53:05.515230 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.515173 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:05.673576 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.673523 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:05.673917 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.673863 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:05.794768 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.794728 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 18:53:05.811433 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.811394 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:05.842924 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.842880 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:05.847629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.844629 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:53:05.975950 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.975847 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 18:53:06.014658 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.986531 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:53:06.014658 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:05.976687 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:06.086654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.082969 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:06.086654 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.083254 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 18:53:06.096705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.087069 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 18:53:06.389786 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.384724 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 18:53:06.409801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.391755 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 18:53:06.466547 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.465905 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 18:53:06.527298 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.526745 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:06.579760 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.579731 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 18:53:06.580807 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.580023 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 18:53:06.683689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.673744 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:06.683689 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.673810 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:06.793553 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.793524 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 18:53:06.803379 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.803355 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 18:53:06.826733 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.826693 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:06.872352 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:06.872322 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 18:53:07.654317 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:07.654286 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:07.654723 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:07.654334 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:08.654833 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:08.654789 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:08.655283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:08.654858 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:09.233247 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:09.233219 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:53:09.654102 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:09.654073 5959 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:53:09.654261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:09.654124 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:53:10.773056 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:10.773026 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:53:10.773569 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:10.773435 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:53:10.774200 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:10.774183 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:53:13.589674 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.589641 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp"] Oct 30 18:53:13.595043 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.595023 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.602899 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.602877 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8"] Oct 30 18:53:13.610241 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.610216 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.622045 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.622020 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp"] Oct 30 18:53:13.630248 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.630223 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8"] Oct 30 18:53:13.711447 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711418 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.711566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711459 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6bl\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-kube-api-access-dz6bl\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.711566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711482 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.711694 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711557 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.711694 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711588 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.711694 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711654 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.711694 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711676 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.711853 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711742 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.711853 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711801 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.711934 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711858 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.711934 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711875 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.712011 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711940 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.712011 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711956 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.712011 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711972 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.712011 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.711986 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.712011 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.712009 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.712488 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.712155 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2qj\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-kube-api-access-wb2qj\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.712488 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.712189 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.812862 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.812832 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.812989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.812869 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6bl\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-kube-api-access-dz6bl\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.812989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.812894 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.812989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.812918 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.813152 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813062 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813152 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813144 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813192 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.813258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813223 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813370 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813314 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.813427 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813366 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.813427 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813391 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.813427 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813390 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813444 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813484 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.813592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813509 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.813592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813536 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.813592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813560 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813590 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813669 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813691 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2qj\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-kube-api-access-wb2qj\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.813928 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.813700 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.814092 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.814002 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.814256 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.814235 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.814419 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.814394 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.814697 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.814633 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.814786 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.814725 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.814786 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.814731 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.815162 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.815045 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.816025 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.815996 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.816521 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.816500 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.817185 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.817164 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.817424 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.817405 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.822240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.822204 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2qj\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-kube-api-access-wb2qj\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.822499 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.822478 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.822663 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.822511 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.822663 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.822548 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6bl\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-kube-api-access-dz6bl\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:13.937362 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.937333 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:13.948457 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:13.948439 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:14.221591 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:14.221569 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp"] Oct 30 18:53:14.223658 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:53:14.223632 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f514dd9_e3d4_4533_b6ba_422964e0ae43.slice/crio-9af1ccd3e5a5475a16f5abdda3b3e328eaf34c31a36c955c2dfb028f1a37988d WatchSource:0}: Error finding container 9af1ccd3e5a5475a16f5abdda3b3e328eaf34c31a36c955c2dfb028f1a37988d: Status 404 returned error can't find the container with id 9af1ccd3e5a5475a16f5abdda3b3e328eaf34c31a36c955c2dfb028f1a37988d Oct 30 18:53:14.226017 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:14.225988 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:14.226080 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:14.226053 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:14.226115 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:14.226081 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:14.233006 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:14.232988 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8"] Oct 30 18:53:14.235964 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:53:14.235938 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb53f1d_da29_4ce3_bd69_19aa0c1dd873.slice/crio-eaa789cbf231c5ab968e5a3cf6059a254966ca0b13068236762cb394d232f9c6 WatchSource:0}: Error finding container eaa789cbf231c5ab968e5a3cf6059a254966ca0b13068236762cb394d232f9c6: Status 404 returned error can't find the container with id eaa789cbf231c5ab968e5a3cf6059a254966ca0b13068236762cb394d232f9c6 Oct 30 18:53:14.238143 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:14.238113 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:14.238237 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:14.238186 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:14.238237 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:14.238221 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:15.226630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.210469 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" event={"ID":"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873","Type":"ContainerStarted","Data":"3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3"} Oct 30 18:53:15.226630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.210555 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" event={"ID":"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873","Type":"ContainerStarted","Data":"eaa789cbf231c5ab968e5a3cf6059a254966ca0b13068236762cb394d232f9c6"} Oct 30 18:53:15.255630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.252673 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podStartSLOduration=2.252654774 podStartE2EDuration="2.252654774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:15.25175154 +0000 UTC m=+975.580162925" watchObservedRunningTime="2026-10-30 18:53:15.252654774 +0000 UTC m=+975.581066160" Oct 30 18:53:15.255630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.254396 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" event={"ID":"2f514dd9-e3d4-4533-b6ba-422964e0ae43","Type":"ContainerStarted","Data":"08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab"} Oct 30 18:53:15.255630 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.254458 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" event={"ID":"2f514dd9-e3d4-4533-b6ba-422964e0ae43","Type":"ContainerStarted","Data":"9af1ccd3e5a5475a16f5abdda3b3e328eaf34c31a36c955c2dfb028f1a37988d"} Oct 30 18:53:15.277041 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.276998 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" podStartSLOduration=2.276985796 podStartE2EDuration="2.276985796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:15.274474873 +0000 UTC m=+975.602886280" watchObservedRunningTime="2026-10-30 18:53:15.276985796 +0000 UTC m=+975.605397175" Oct 30 18:53:15.941322 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.941291 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:15.941727 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.941706 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:53:15.941922 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.941895 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:53:15.956947 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.956916 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:15.957352 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.957332 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:53:15.957525 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:15.957504 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:53:16.938080 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:16.938042 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:53:16.938730 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:16.938699 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:53:16.949214 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:16.949185 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:53:16.949385 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:16.949239 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:53:17.938061 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:17.938023 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:53:17.938362 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:17.938333 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:53:17.949711 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:17.949683 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:53:17.949830 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:17.949735 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:53:18.942035 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:18.941995 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:53:18.942672 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:18.942638 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:53:18.950653 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:18.950586 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:53:18.950795 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:18.950659 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:53:19.946452 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:19.946407 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:53:19.947007 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:19.946485 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:53:19.949875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:19.949840 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:53:19.950022 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:19.949901 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:53:20.937774 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:20.937732 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:53:20.937989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:20.937811 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:53:20.948764 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:20.948734 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:53:20.949233 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:20.948771 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:53:22.164258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:22.164230 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:22.164989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:22.164961 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:22.165916 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:22.165900 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 18:53:22.259239 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:22.257547 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:22.259239 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:22.258666 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:22.266651 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:22.259965 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:53:27.780315 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:53:27.780275 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:53:28.103090 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.103012 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8"] Oct 30 18:53:28.108234 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.108209 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:28.113437 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.113410 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7784e5-dockercfg-m2h9z\"" Oct 30 18:53:28.113437 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.113423 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7784e5\"" Oct 30 18:53:28.113876 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.113854 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8"] Oct 30 18:53:28.125642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.125596 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk"] Oct 30 18:53:28.131187 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.130824 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:28.137914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.137887 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk"] Oct 30 18:53:28.232633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.231490 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndvmn\" (UniqueName: \"kubernetes.io/projected/462fad36-1de9-4523-90a8-2ef740e5f7af-kube-api-access-ndvmn\") pod \"payload-processing-e2e-api-b-7784e5-79887659d4-5hll8\" (UID: \"462fad36-1de9-4523-90a8-2ef740e5f7af\") " pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:28.232633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.231639 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d114623c-5b11-4629-87c3-9ff8974cd0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk\" (UID: \"d114623c-5b11-4629-87c3-9ff8974cd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:28.232633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.231902 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462fad36-1de9-4523-90a8-2ef740e5f7af-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7784e5-79887659d4-5hll8\" (UID: \"462fad36-1de9-4523-90a8-2ef740e5f7af\") " pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:28.232633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.231968 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjp2\" (UniqueName: \"kubernetes.io/projected/d114623c-5b11-4629-87c3-9ff8974cd0ad-kube-api-access-xrjp2\") pod \"payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk\" (UID: \"d114623c-5b11-4629-87c3-9ff8974cd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:28.352596 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.352562 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndvmn\" (UniqueName: \"kubernetes.io/projected/462fad36-1de9-4523-90a8-2ef740e5f7af-kube-api-access-ndvmn\") pod \"payload-processing-e2e-api-b-7784e5-79887659d4-5hll8\" (UID: \"462fad36-1de9-4523-90a8-2ef740e5f7af\") " pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:28.352790 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.352680 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d114623c-5b11-4629-87c3-9ff8974cd0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk\" (UID: \"d114623c-5b11-4629-87c3-9ff8974cd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:28.352851 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.352820 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462fad36-1de9-4523-90a8-2ef740e5f7af-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7784e5-79887659d4-5hll8\" (UID: \"462fad36-1de9-4523-90a8-2ef740e5f7af\") " pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:28.352893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.352877 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjp2\" (UniqueName: \"kubernetes.io/projected/d114623c-5b11-4629-87c3-9ff8974cd0ad-kube-api-access-xrjp2\") pod \"payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk\" (UID: \"d114623c-5b11-4629-87c3-9ff8974cd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:28.353455 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.353399 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d114623c-5b11-4629-87c3-9ff8974cd0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk\" (UID: \"d114623c-5b11-4629-87c3-9ff8974cd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:28.353549 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.353481 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462fad36-1de9-4523-90a8-2ef740e5f7af-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7784e5-79887659d4-5hll8\" (UID: \"462fad36-1de9-4523-90a8-2ef740e5f7af\") " pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:28.360905 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.360878 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjp2\" (UniqueName: \"kubernetes.io/projected/d114623c-5b11-4629-87c3-9ff8974cd0ad-kube-api-access-xrjp2\") pod \"payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk\" (UID: \"d114623c-5b11-4629-87c3-9ff8974cd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:28.361031 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.360945 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndvmn\" (UniqueName: \"kubernetes.io/projected/462fad36-1de9-4523-90a8-2ef740e5f7af-kube-api-access-ndvmn\") pod \"payload-processing-e2e-api-b-7784e5-79887659d4-5hll8\" (UID: \"462fad36-1de9-4523-90a8-2ef740e5f7af\") " pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:28.426836 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.426805 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:28.450359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.450330 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:28.712563 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.712539 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8"] Oct 30 18:53:28.713296 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:53:28.713270 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462fad36_1de9_4523_90a8_2ef740e5f7af.slice/crio-49bb1ce6a2c0645aaa16b36f5d99b316cd07d9a5049dd89b90eea08fb67006e8 WatchSource:0}: Error finding container 49bb1ce6a2c0645aaa16b36f5d99b316cd07d9a5049dd89b90eea08fb67006e8: Status 404 returned error can't find the container with id 49bb1ce6a2c0645aaa16b36f5d99b316cd07d9a5049dd89b90eea08fb67006e8 Oct 30 18:53:28.774520 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:28.774500 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk"] Oct 30 18:53:28.776566 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:53:28.776535 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd114623c_5b11_4629_87c3_9ff8974cd0ad.slice/crio-4ec6f40214ef435badde326e64cd43b6983e4b5bf5f3b5c0af701a3dddf3d0ed WatchSource:0}: Error finding container 4ec6f40214ef435badde326e64cd43b6983e4b5bf5f3b5c0af701a3dddf3d0ed: Status 404 returned error can't find the container with id 4ec6f40214ef435badde326e64cd43b6983e4b5bf5f3b5c0af701a3dddf3d0ed Oct 30 18:53:29.435581 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:29.435543 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" event={"ID":"462fad36-1de9-4523-90a8-2ef740e5f7af","Type":"ContainerStarted","Data":"2d8cd4062ba17ec31e1140801698ed0fc178550bd91bbd56fad37efc6fafc8ff"} Oct 30 18:53:29.436046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:29.435631 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:29.436046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:29.435659 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" event={"ID":"462fad36-1de9-4523-90a8-2ef740e5f7af","Type":"ContainerStarted","Data":"49bb1ce6a2c0645aaa16b36f5d99b316cd07d9a5049dd89b90eea08fb67006e8"} Oct 30 18:53:29.437169 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:29.437145 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" event={"ID":"d114623c-5b11-4629-87c3-9ff8974cd0ad","Type":"ContainerStarted","Data":"d9186c4f0a64330e5954277d5331675fbf5fa705bc6b63bb8d9f6ffd838b2011"} Oct 30 18:53:29.437311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:29.437190 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" event={"ID":"d114623c-5b11-4629-87c3-9ff8974cd0ad","Type":"ContainerStarted","Data":"4ec6f40214ef435badde326e64cd43b6983e4b5bf5f3b5c0af701a3dddf3d0ed"} Oct 30 18:53:29.437311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:29.437221 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:29.453948 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:29.453907 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" podStartSLOduration=1.453894 podStartE2EDuration="1.453894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:29.451682014 +0000 UTC m=+989.780093400" watchObservedRunningTime="2026-10-30 18:53:29.453894 +0000 UTC m=+989.782305384" Oct 30 18:53:29.468440 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:29.468394 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" podStartSLOduration=1.468382109 podStartE2EDuration="1.468382109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:29.468220606 +0000 UTC m=+989.796631991" watchObservedRunningTime="2026-10-30 18:53:29.468382109 +0000 UTC m=+989.796793495" Oct 30 18:53:34.371058 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:34.371017 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:53:35.851106 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.851076 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns"] Oct 30 18:53:35.857057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.857032 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss"] Oct 30 18:53:35.857193 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.857174 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.863062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.861491 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.871513 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.871075 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss"] Oct 30 18:53:35.874941 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.874915 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns"] Oct 30 18:53:35.914339 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914307 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.914497 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914368 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.914497 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914474 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.914695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914516 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.914695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914560 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2a48a4-f066-4521-9150-8131ed67405d-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.914695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914635 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj58b\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-kube-api-access-wj58b\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.914695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914678 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.914695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914694 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914710 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914747 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914773 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914795 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914812 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnnw\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-kube-api-access-4cnnw\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914851 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2a48a4-f066-4521-9150-8131ed67405d-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914871 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914926 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914978 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:35.915046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.914998 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:35.943513 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:35.943492 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f37d42-openshift-default-dockercfg-9hc5n\"" Oct 30 18:53:36.016138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016108 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj58b\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-kube-api-access-wj58b\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016324 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016169 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016324 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016201 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.016324 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016223 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.016324 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016272 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.016324 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016311 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016343 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016390 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnnw\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-kube-api-access-4cnnw\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.016640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016511 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2a48a4-f066-4521-9150-8131ed67405d-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.016640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016545 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.016640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016619 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016666 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016691 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.016901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016693 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016700 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.016901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016777 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016801 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.016901 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016872 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.017258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016922 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.017258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016939 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.017258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.016974 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.017258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.017009 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.017258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.017051 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2a48a4-f066-4521-9150-8131ed67405d-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.017258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.017058 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.017258 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.017200 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.017532 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.017358 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.017532 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.017433 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2a48a4-f066-4521-9150-8131ed67405d-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.017670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.017554 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.019045 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.019016 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.019465 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.019442 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.019782 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.019755 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.019972 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.019957 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2a48a4-f066-4521-9150-8131ed67405d-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.024566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.024546 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnnw\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-kube-api-access-4cnnw\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.025249 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.025220 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj58b\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-kube-api-access-wj58b\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.025348 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.025244 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.025725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.025687 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.182991 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.182957 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:36.193383 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.193356 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:36.489217 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.489156 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss"] Oct 30 18:53:36.492173 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.492145 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns"] Oct 30 18:53:36.492667 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:53:36.492639 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2a48a4_f066_4521_9150_8131ed67405d.slice/crio-102727f5845a832b046fdbc38b9b92c7422550f287c3b263ed04414e21cd2bb1 WatchSource:0}: Error finding container 102727f5845a832b046fdbc38b9b92c7422550f287c3b263ed04414e21cd2bb1: Status 404 returned error can't find the container with id 102727f5845a832b046fdbc38b9b92c7422550f287c3b263ed04414e21cd2bb1 Oct 30 18:53:36.493797 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:53:36.493778 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd700bcc8_04d2_45ef_b3c3_ddb9382d71a4.slice/crio-13b6f325e6a1e1c08612b1d737770e8355b4cdf369028f804986f82ea64ed2fd WatchSource:0}: Error finding container 13b6f325e6a1e1c08612b1d737770e8355b4cdf369028f804986f82ea64ed2fd: Status 404 returned error can't find the container with id 13b6f325e6a1e1c08612b1d737770e8355b4cdf369028f804986f82ea64ed2fd Oct 30 18:53:36.495677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.495643 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:36.495755 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.495721 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:36.495825 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.495763 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:36.496237 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.496211 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:36.496315 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.496266 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:36.496315 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:36.496295 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:37.491430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:37.491397 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" event={"ID":"7e2a48a4-f066-4521-9150-8131ed67405d","Type":"ContainerStarted","Data":"c34e768e6b96b2603e873fb8fb8d3af991c43dc26cae48b8021a46e765958d63"} Oct 30 18:53:37.491863 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:37.491450 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" event={"ID":"7e2a48a4-f066-4521-9150-8131ed67405d","Type":"ContainerStarted","Data":"102727f5845a832b046fdbc38b9b92c7422550f287c3b263ed04414e21cd2bb1"} Oct 30 18:53:37.493960 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:37.493936 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" event={"ID":"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4","Type":"ContainerStarted","Data":"051cee7a3811f41829b9c869904b721a6a9420af9a839913b391626050633303"} Oct 30 18:53:37.494103 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:37.493986 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" event={"ID":"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4","Type":"ContainerStarted","Data":"13b6f325e6a1e1c08612b1d737770e8355b4cdf369028f804986f82ea64ed2fd"} Oct 30 18:53:37.571372 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:37.571319 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" podStartSLOduration=2.57130375 podStartE2EDuration="2.57130375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:37.550118954 +0000 UTC m=+997.878530340" watchObservedRunningTime="2026-10-30 18:53:37.57130375 +0000 UTC m=+997.899715134" Oct 30 18:53:37.573162 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:37.573128 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" podStartSLOduration=2.573116246 podStartE2EDuration="2.573116246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:37.570477969 +0000 UTC m=+997.898889353" watchObservedRunningTime="2026-10-30 18:53:37.573116246 +0000 UTC m=+997.901527622" Oct 30 18:53:38.184083 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:38.184052 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:38.185749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:38.185726 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:53:38.185815 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:38.185778 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:53:38.193661 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:38.193646 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:38.194747 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:38.194725 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:53:38.194840 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:38.194763 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:53:39.183510 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:39.183472 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:53:39.183937 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:39.183532 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:53:39.194440 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:39.194418 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:53:39.194539 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:39.194467 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:53:40.184311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:40.184277 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:53:40.184741 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:40.184329 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:53:40.204793 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:40.204753 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:53:40.205079 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:40.205045 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:53:40.462897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:40.462781 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:53:40.463793 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:40.463773 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:53:41.183930 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:41.183894 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:53:41.184109 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:41.183946 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:53:41.194354 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:41.194326 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:53:41.194880 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:41.194853 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:53:42.183713 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:42.183674 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:53:42.183914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:42.183745 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:53:42.194412 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:42.194386 5959 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:53:42.194801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:42.194435 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:53:43.187622 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:43.187579 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:43.188035 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:43.188016 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:43.188699 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:43.188682 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:53:43.198253 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:43.198234 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:43.198576 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:43.198474 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:43.199081 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:43.199066 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:53:44.638457 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.638424 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z"] Oct 30 18:53:44.643341 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.643320 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:44.659782 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.659754 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-da58b769-dockercfg-dpkmr\"" Oct 30 18:53:44.663657 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.663625 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z"] Oct 30 18:53:44.689677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.689644 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577a654a-8c33-42c0-a3c5-ade3831ab907-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z\" (UID: \"577a654a-8c33-42c0-a3c5-ade3831ab907\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:44.689917 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.689895 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnj9\" (UniqueName: \"kubernetes.io/projected/577a654a-8c33-42c0-a3c5-ade3831ab907-kube-api-access-2fnj9\") pod \"payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z\" (UID: \"577a654a-8c33-42c0-a3c5-ade3831ab907\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:44.715494 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.715471 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-da58b769\"" Oct 30 18:53:44.791417 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.791377 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577a654a-8c33-42c0-a3c5-ade3831ab907-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z\" (UID: \"577a654a-8c33-42c0-a3c5-ade3831ab907\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:44.807632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.804095 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnj9\" (UniqueName: \"kubernetes.io/projected/577a654a-8c33-42c0-a3c5-ade3831ab907-kube-api-access-2fnj9\") pod \"payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z\" (UID: \"577a654a-8c33-42c0-a3c5-ade3831ab907\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:44.821625 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.803847 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577a654a-8c33-42c0-a3c5-ade3831ab907-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z\" (UID: \"577a654a-8c33-42c0-a3c5-ade3831ab907\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:44.821873 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.818453 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnj9\" (UniqueName: \"kubernetes.io/projected/577a654a-8c33-42c0-a3c5-ade3831ab907-kube-api-access-2fnj9\") pod \"payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z\" (UID: \"577a654a-8c33-42c0-a3c5-ade3831ab907\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:44.968137 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:44.968051 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:45.255074 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:45.254930 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z"] Oct 30 18:53:45.257514 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:53:45.257482 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577a654a_8c33_42c0_a3c5_ade3831ab907.slice/crio-3f2d7435663fa78f14def2e1c51367ebbdb1e2388fd07d698cfeb6ae26ac86b1 WatchSource:0}: Error finding container 3f2d7435663fa78f14def2e1c51367ebbdb1e2388fd07d698cfeb6ae26ac86b1: Status 404 returned error can't find the container with id 3f2d7435663fa78f14def2e1c51367ebbdb1e2388fd07d698cfeb6ae26ac86b1 Oct 30 18:53:45.643315 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:45.643276 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" event={"ID":"577a654a-8c33-42c0-a3c5-ade3831ab907","Type":"ContainerStarted","Data":"60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6"} Oct 30 18:53:45.643725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:45.643339 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" event={"ID":"577a654a-8c33-42c0-a3c5-ade3831ab907","Type":"ContainerStarted","Data":"3f2d7435663fa78f14def2e1c51367ebbdb1e2388fd07d698cfeb6ae26ac86b1"} Oct 30 18:53:45.643725 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:45.643367 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:45.664138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:45.664081 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" podStartSLOduration=1.664062794 podStartE2EDuration="1.664062794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:45.659475161 +0000 UTC m=+1005.987886543" watchObservedRunningTime="2026-10-30 18:53:45.664062794 +0000 UTC m=+1005.992474180" Oct 30 18:53:52.921596 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:52.921557 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj"] Oct 30 18:53:52.925890 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:52.925871 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:52.929782 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:52.929759 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dqnb2\"" Oct 30 18:53:52.932274 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:52.932254 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:53:52.932450 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:52.932437 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:53:52.932510 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:52.932452 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:53:52.938364 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:52.938340 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj"] Oct 30 18:53:53.044103 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.044074 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.044296 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.044112 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8hf2\" (UniqueName: \"kubernetes.io/projected/5807139c-f7d5-4f02-a866-88a6cb783af9-kube-api-access-r8hf2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.044368 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.044290 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.044430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.044395 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5807139c-f7d5-4f02-a866-88a6cb783af9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.044541 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.044524 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.044600 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.044554 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.044671 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.044658 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.145488 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.145450 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.145715 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.145524 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5807139c-f7d5-4f02-a866-88a6cb783af9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.145715 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.145620 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.145715 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.145644 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.145866 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.145801 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.145924 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.145889 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.145982 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.145932 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8hf2\" (UniqueName: \"kubernetes.io/projected/5807139c-f7d5-4f02-a866-88a6cb783af9-kube-api-access-r8hf2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.146884 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.146857 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.148189 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.148157 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.148302 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.148244 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.148449 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.148431 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5807139c-f7d5-4f02-a866-88a6cb783af9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.148492 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.148477 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5807139c-f7d5-4f02-a866-88a6cb783af9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.153794 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.153768 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8hf2\" (UniqueName: \"kubernetes.io/projected/5807139c-f7d5-4f02-a866-88a6cb783af9-kube-api-access-r8hf2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.153794 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.153785 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5807139c-f7d5-4f02-a866-88a6cb783af9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gs6dj\" (UID: \"5807139c-f7d5-4f02-a866-88a6cb783af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.241165 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.241062 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:53.536360 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.536335 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj"] Oct 30 18:53:53.538161 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:53:53.538133 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5807139c_f7d5_4f02_a866_88a6cb783af9.slice/crio-d1e8db9950c1afebf6131eba1026ba8fb51d11b35e9d62a738d90fd0c5014cac WatchSource:0}: Error finding container d1e8db9950c1afebf6131eba1026ba8fb51d11b35e9d62a738d90fd0c5014cac: Status 404 returned error can't find the container with id d1e8db9950c1afebf6131eba1026ba8fb51d11b35e9d62a738d90fd0c5014cac Oct 30 18:53:53.540353 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.540339 5959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:53:53.701059 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:53.701031 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" event={"ID":"5807139c-f7d5-4f02-a866-88a6cb783af9","Type":"ContainerStarted","Data":"d1e8db9950c1afebf6131eba1026ba8fb51d11b35e9d62a738d90fd0c5014cac"} Oct 30 18:53:56.191065 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:56.191025 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:56.191583 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:56.191196 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:53:56.655718 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:56.655687 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:53:56.738699 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:56.738649 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" event={"ID":"5807139c-f7d5-4f02-a866-88a6cb783af9","Type":"ContainerStarted","Data":"97f4912cd2b24a40b3da2b623bff3252d2f1395da8f9e7b2ba5244cb10da1172"} Oct 30 18:53:56.738943 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:56.738788 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:56.764468 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:56.764405 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" podStartSLOduration=2.115597908 podStartE2EDuration="4.764382797s" totalImagesPullingTime="2.648784889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:56.759372938 +0000 UTC m=+1017.087784334" watchObservedRunningTime="2026-10-30 18:53:56.764382797 +0000 UTC m=+1017.092794189" Oct 30 18:53:56.980677 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:56.963253 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn"] Oct 30 18:53:57.747329 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:53:57.747302 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gs6dj" Oct 30 18:53:57.782731 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:53:57.782689 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:54:01.963985 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:01.963942 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" podUID="daea56e1-2b7e-45dd-8554-727a7546d84c" containerName="payload-pre-processing" containerID="cri-o://cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc" gracePeriod=25 Oct 30 18:54:02.238387 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.238324 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:54:02.354720 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.354689 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daea56e1-2b7e-45dd-8554-727a7546d84c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daea56e1-2b7e-45dd-8554-727a7546d84c-plugins-config-volume\") pod \"daea56e1-2b7e-45dd-8554-727a7546d84c\" (UID: \"daea56e1-2b7e-45dd-8554-727a7546d84c\") " Oct 30 18:54:02.354885 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.354753 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daea56e1-2b7e-45dd-8554-727a7546d84c-kube-api-access-cmxvd\" (UniqueName: \"kubernetes.io/projected/daea56e1-2b7e-45dd-8554-727a7546d84c-kube-api-access-cmxvd\") pod \"daea56e1-2b7e-45dd-8554-727a7546d84c\" (UID: \"daea56e1-2b7e-45dd-8554-727a7546d84c\") " Oct 30 18:54:02.355127 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.355102 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daea56e1-2b7e-45dd-8554-727a7546d84c-plugins-config-volume" pod "daea56e1-2b7e-45dd-8554-727a7546d84c" (UID: "daea56e1-2b7e-45dd-8554-727a7546d84c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:02.355348 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.355334 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daea56e1-2b7e-45dd-8554-727a7546d84c-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:02.357047 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.357027 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daea56e1-2b7e-45dd-8554-727a7546d84c-kube-api-access-cmxvd" pod "daea56e1-2b7e-45dd-8554-727a7546d84c" (UID: "daea56e1-2b7e-45dd-8554-727a7546d84c"). InnerVolumeSpecName "kube-api-access-cmxvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:02.456748 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.456719 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxvd\" (UniqueName: \"kubernetes.io/projected/daea56e1-2b7e-45dd-8554-727a7546d84c-kube-api-access-cmxvd\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:02.788914 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.788878 5959 generic.go:356] "Generic (PLEG): container finished" podID="daea56e1-2b7e-45dd-8554-727a7546d84c" containerID="cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc" exitCode=0 Oct 30 18:54:02.789115 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.788929 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" Oct 30 18:54:02.789115 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.788960 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" event={"ID":"daea56e1-2b7e-45dd-8554-727a7546d84c","Type":"ContainerDied","Data":"cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc"} Oct 30 18:54:02.789115 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.789013 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn" event={"ID":"daea56e1-2b7e-45dd-8554-727a7546d84c","Type":"ContainerDied","Data":"c74de60a4e41ba7ff4a7922916801af6bc96ef73e8a7c6a5b6ecca49897dca5e"} Oct 30 18:54:02.789115 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.789082 5959 scope.go:122] "RemoveContainer" containerID="cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc" Oct 30 18:54:02.800261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.800243 5959 scope.go:122] "RemoveContainer" containerID="cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc" Oct 30 18:54:02.800523 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:54:02.800503 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc\": container with ID starting with cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc not found: ID does not exist" containerID="cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc" Oct 30 18:54:02.800599 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.800534 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc"} err="failed to get container status \"cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc\": rpc error: code = NotFound desc = could not find container \"cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc\": container with ID starting with cf14c9f5b19617b7021930f27f8cf85a350ac8ac87d98a6586e08fe02c023edc not found: ID does not exist" Oct 30 18:54:02.813792 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.813773 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn"] Oct 30 18:54:02.816582 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:02.816564 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87293f-58bf6f75cf-85cbn"] Oct 30 18:54:03.801800 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:03.801762 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daea56e1-2b7e-45dd-8554-727a7546d84c" path="/var/lib/kubelet/pods/daea56e1-2b7e-45dd-8554-727a7546d84c/volumes" Oct 30 18:54:08.552270 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:08.552235 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:54:19.728688 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:19.728642 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns"] Oct 30 18:54:19.733121 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:19.733087 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss"] Oct 30 18:54:19.733539 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:19.733488 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" containerID="cri-o://c34e768e6b96b2603e873fb8fb8d3af991c43dc26cae48b8021a46e765958d63" gracePeriod=30 Oct 30 18:54:19.733828 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:19.733234 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" containerID="cri-o://051cee7a3811f41829b9c869904b721a6a9420af9a839913b391626050633303" gracePeriod=30 Oct 30 18:54:25.010019 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.009941 5959 generic.go:356] "Generic (PLEG): container finished" podID="7e2a48a4-f066-4521-9150-8131ed67405d" containerID="c34e768e6b96b2603e873fb8fb8d3af991c43dc26cae48b8021a46e765958d63" exitCode=0 Oct 30 18:54:25.010019 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.010001 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" event={"ID":"7e2a48a4-f066-4521-9150-8131ed67405d","Type":"ContainerDied","Data":"c34e768e6b96b2603e873fb8fb8d3af991c43dc26cae48b8021a46e765958d63"} Oct 30 18:54:25.011978 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.011957 5959 generic.go:356] "Generic (PLEG): container finished" podID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerID="051cee7a3811f41829b9c869904b721a6a9420af9a839913b391626050633303" exitCode=0 Oct 30 18:54:25.012104 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.012030 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" event={"ID":"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4","Type":"ContainerDied","Data":"051cee7a3811f41829b9c869904b721a6a9420af9a839913b391626050633303"} Oct 30 18:54:25.012104 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.012081 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" event={"ID":"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4","Type":"ContainerDied","Data":"13b6f325e6a1e1c08612b1d737770e8355b4cdf369028f804986f82ea64ed2fd"} Oct 30 18:54:25.012215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.012122 5959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b6f325e6a1e1c08612b1d737770e8355b4cdf369028f804986f82ea64ed2fd" Oct 30 18:54:25.032422 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.032404 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:54:25.062123 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.062050 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:54:25.143849 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.143822 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-token\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-token\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.144098 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.143896 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-certs\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.144098 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144050 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-data\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.144220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144101 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-credential-socket\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.144220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144120 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-credential-socket\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.144220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144124 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-certs" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.144220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144152 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-socket\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.144220 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144191 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-envoy\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.144480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144245 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-data\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.144480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144309 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-podinfo\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.144480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144343 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-credential-socket" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.144480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144353 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-data" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.144480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144382 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-credential-socket" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.144480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144418 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-socket" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.144480 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144452 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-certs\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144491 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-kube-api-access-wj58b\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-kube-api-access-wj58b\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144530 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e2a48a4-f066-4521-9150-8131ed67405d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2a48a4-f066-4521-9150-8131ed67405d-istio-podinfo\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144560 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-socket\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144561 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-data" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144687 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istiod-ca-cert\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144761 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2a48a4-f066-4521-9150-8131ed67405d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2a48a4-f066-4521-9150-8131ed67405d-istiod-ca-cert\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144820 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-socket" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144834 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-istio-token\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-istio-token\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.144869 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144868 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-envoy\") pod \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\" (UID: \"d700bcc8-04d2-45ef-b3c3-ddb9382d71a4\") " Oct 30 18:54:25.145283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.144919 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-kube-api-access-4cnnw\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-kube-api-access-4cnnw\") pod \"7e2a48a4-f066-4521-9150-8131ed67405d\" (UID: \"7e2a48a4-f066-4521-9150-8131ed67405d\") " Oct 30 18:54:25.145283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145269 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-certs" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.145386 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145350 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istiod-ca-cert" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145716 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145743 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145758 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145772 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145787 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145800 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145814 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145828 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.145860 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.145841 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.146865 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.146836 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-token" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:25.147148 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.147124 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-podinfo" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:54:25.147260 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.147158 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2a48a4-f066-4521-9150-8131ed67405d-istiod-ca-cert" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:25.147260 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.147239 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e2a48a4-f066-4521-9150-8131ed67405d-istio-podinfo" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:54:25.147573 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.147550 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-envoy" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.147877 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.147860 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-kube-api-access-4cnnw" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "kube-api-access-4cnnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:25.148169 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.148149 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-kube-api-access-wj58b" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "kube-api-access-wj58b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:25.148264 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.148229 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-envoy" pod "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" (UID: "d700bcc8-04d2-45ef-b3c3-ddb9382d71a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:54:25.148449 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.148437 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-istio-token" pod "7e2a48a4-f066-4521-9150-8131ed67405d" (UID: "7e2a48a4-f066-4521-9150-8131ed67405d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:25.246789 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246762 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj58b\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-kube-api-access-wj58b\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.246789 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246786 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2a48a4-f066-4521-9150-8131ed67405d-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.246989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246798 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2a48a4-f066-4521-9150-8131ed67405d-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.246989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246812 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.246989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246823 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.246989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246831 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnnw\" (UniqueName: \"kubernetes.io/projected/7e2a48a4-f066-4521-9150-8131ed67405d-kube-api-access-4cnnw\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.246989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246839 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.246989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246849 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2a48a4-f066-4521-9150-8131ed67405d-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:25.246989 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:25.246857 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:26.019959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:26.019866 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" event={"ID":"7e2a48a4-f066-4521-9150-8131ed67405d","Type":"ContainerDied","Data":"102727f5845a832b046fdbc38b9b92c7422550f287c3b263ed04414e21cd2bb1"} Oct 30 18:54:26.019959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:26.019919 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss" Oct 30 18:54:26.019959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:26.019937 5959 scope.go:122] "RemoveContainer" containerID="c34e768e6b96b2603e873fb8fb8d3af991c43dc26cae48b8021a46e765958d63" Oct 30 18:54:26.020466 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:26.019919 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns" Oct 30 18:54:26.042792 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:26.042715 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss"] Oct 30 18:54:26.046436 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:26.046414 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-6xbss"] Oct 30 18:54:26.056749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:26.056730 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns"] Oct 30 18:54:26.060538 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:26.060518 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-jxwns"] Oct 30 18:54:27.779989 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:54:27.779956 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:54:27.798339 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:27.798304 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" path="/var/lib/kubelet/pods/7e2a48a4-f066-4521-9150-8131ed67405d/volumes" Oct 30 18:54:27.798965 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:27.798949 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" path="/var/lib/kubelet/pods/d700bcc8-04d2-45ef-b3c3-ddb9382d71a4/volumes" Oct 30 18:54:28.808042 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:28.808013 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8"] Oct 30 18:54:28.832886 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:28.832858 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk"] Oct 30 18:54:33.808986 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:33.808933 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" podUID="462fad36-1de9-4523-90a8-2ef740e5f7af" containerName="payload-processing" containerID="cri-o://2d8cd4062ba17ec31e1140801698ed0fc178550bd91bbd56fad37efc6fafc8ff" gracePeriod=25 Oct 30 18:54:33.833704 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:33.833670 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" podUID="d114623c-5b11-4629-87c3-9ff8974cd0ad" containerName="payload-pre-processing" containerID="cri-o://d9186c4f0a64330e5954277d5331675fbf5fa705bc6b63bb8d9f6ffd838b2011" gracePeriod=25 Oct 30 18:54:34.090542 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.090511 5959 generic.go:356] "Generic (PLEG): container finished" podID="462fad36-1de9-4523-90a8-2ef740e5f7af" containerID="2d8cd4062ba17ec31e1140801698ed0fc178550bd91bbd56fad37efc6fafc8ff" exitCode=0 Oct 30 18:54:34.090774 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.090743 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" event={"ID":"462fad36-1de9-4523-90a8-2ef740e5f7af","Type":"ContainerDied","Data":"2d8cd4062ba17ec31e1140801698ed0fc178550bd91bbd56fad37efc6fafc8ff"} Oct 30 18:54:34.093166 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.093133 5959 generic.go:356] "Generic (PLEG): container finished" podID="d114623c-5b11-4629-87c3-9ff8974cd0ad" containerID="d9186c4f0a64330e5954277d5331675fbf5fa705bc6b63bb8d9f6ffd838b2011" exitCode=0 Oct 30 18:54:34.093432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.093181 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" event={"ID":"d114623c-5b11-4629-87c3-9ff8974cd0ad","Type":"ContainerDied","Data":"d9186c4f0a64330e5954277d5331675fbf5fa705bc6b63bb8d9f6ffd838b2011"} Oct 30 18:54:34.128964 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.128938 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:54:34.169449 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.169424 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:54:34.181487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.181434 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d114623c-5b11-4629-87c3-9ff8974cd0ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d114623c-5b11-4629-87c3-9ff8974cd0ad-plugins-config-volume\") pod \"d114623c-5b11-4629-87c3-9ff8974cd0ad\" (UID: \"d114623c-5b11-4629-87c3-9ff8974cd0ad\") " Oct 30 18:54:34.181703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.181554 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462fad36-1de9-4523-90a8-2ef740e5f7af-kube-api-access-ndvmn\" (UniqueName: \"kubernetes.io/projected/462fad36-1de9-4523-90a8-2ef740e5f7af-kube-api-access-ndvmn\") pod \"462fad36-1de9-4523-90a8-2ef740e5f7af\" (UID: \"462fad36-1de9-4523-90a8-2ef740e5f7af\") " Oct 30 18:54:34.181703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.181674 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462fad36-1de9-4523-90a8-2ef740e5f7af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462fad36-1de9-4523-90a8-2ef740e5f7af-plugins-config-volume\") pod \"462fad36-1de9-4523-90a8-2ef740e5f7af\" (UID: \"462fad36-1de9-4523-90a8-2ef740e5f7af\") " Oct 30 18:54:34.181833 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.181776 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d114623c-5b11-4629-87c3-9ff8974cd0ad-kube-api-access-xrjp2\" (UniqueName: \"kubernetes.io/projected/d114623c-5b11-4629-87c3-9ff8974cd0ad-kube-api-access-xrjp2\") pod \"d114623c-5b11-4629-87c3-9ff8974cd0ad\" (UID: \"d114623c-5b11-4629-87c3-9ff8974cd0ad\") " Oct 30 18:54:34.182540 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.182462 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d114623c-5b11-4629-87c3-9ff8974cd0ad-plugins-config-volume" pod "d114623c-5b11-4629-87c3-9ff8974cd0ad" (UID: "d114623c-5b11-4629-87c3-9ff8974cd0ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:34.182540 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.182498 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462fad36-1de9-4523-90a8-2ef740e5f7af-plugins-config-volume" pod "462fad36-1de9-4523-90a8-2ef740e5f7af" (UID: "462fad36-1de9-4523-90a8-2ef740e5f7af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:34.184871 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.184842 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d114623c-5b11-4629-87c3-9ff8974cd0ad-kube-api-access-xrjp2" pod "d114623c-5b11-4629-87c3-9ff8974cd0ad" (UID: "d114623c-5b11-4629-87c3-9ff8974cd0ad"). InnerVolumeSpecName "kube-api-access-xrjp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:34.184993 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.184975 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462fad36-1de9-4523-90a8-2ef740e5f7af-kube-api-access-ndvmn" pod "462fad36-1de9-4523-90a8-2ef740e5f7af" (UID: "462fad36-1de9-4523-90a8-2ef740e5f7af"). InnerVolumeSpecName "kube-api-access-ndvmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:34.283476 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.283436 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d114623c-5b11-4629-87c3-9ff8974cd0ad-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:34.283476 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.283468 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndvmn\" (UniqueName: \"kubernetes.io/projected/462fad36-1de9-4523-90a8-2ef740e5f7af-kube-api-access-ndvmn\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:34.283476 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.283478 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462fad36-1de9-4523-90a8-2ef740e5f7af-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:34.283756 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:34.283487 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjp2\" (UniqueName: \"kubernetes.io/projected/d114623c-5b11-4629-87c3-9ff8974cd0ad-kube-api-access-xrjp2\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:54:35.103753 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.103670 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" Oct 30 18:54:35.104198 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.103666 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk" event={"ID":"d114623c-5b11-4629-87c3-9ff8974cd0ad","Type":"ContainerDied","Data":"4ec6f40214ef435badde326e64cd43b6983e4b5bf5f3b5c0af701a3dddf3d0ed"} Oct 30 18:54:35.104198 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.103874 5959 scope.go:122] "RemoveContainer" containerID="d9186c4f0a64330e5954277d5331675fbf5fa705bc6b63bb8d9f6ffd838b2011" Oct 30 18:54:35.108189 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.108160 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" event={"ID":"462fad36-1de9-4523-90a8-2ef740e5f7af","Type":"ContainerDied","Data":"49bb1ce6a2c0645aaa16b36f5d99b316cd07d9a5049dd89b90eea08fb67006e8"} Oct 30 18:54:35.108288 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.108267 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8" Oct 30 18:54:35.119503 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.119483 5959 scope.go:122] "RemoveContainer" containerID="2d8cd4062ba17ec31e1140801698ed0fc178550bd91bbd56fad37efc6fafc8ff" Oct 30 18:54:35.130539 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.130461 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk"] Oct 30 18:54:35.134568 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.134539 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7784e5-65dbcf8555-fgbxk"] Oct 30 18:54:35.143501 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.143479 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8"] Oct 30 18:54:35.147074 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.147041 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7784e5-79887659d4-5hll8"] Oct 30 18:54:35.797685 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.797655 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462fad36-1de9-4523-90a8-2ef740e5f7af" path="/var/lib/kubelet/pods/462fad36-1de9-4523-90a8-2ef740e5f7af/volumes" Oct 30 18:54:35.798122 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:35.798106 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d114623c-5b11-4629-87c3-9ff8974cd0ad" path="/var/lib/kubelet/pods/d114623c-5b11-4629-87c3-9ff8974cd0ad/volumes" Oct 30 18:54:42.490487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:54:42.490447 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:54:57.783437 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:54:57.783397 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:55:00.540375 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:55:00.540346 5959 scope.go:122] "RemoveContainer" containerID="c3966c6486c90df307511f1dd629e2ded354cd13010367bbfde84513c14e3f1d" Oct 30 18:55:27.662259 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:55:27.662167 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:55:27.866925 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:55:27.866865 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:55:37.159464 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:55:37.159429 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z"] Oct 30 18:55:42.160545 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:55:42.160490 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" podUID="577a654a-8c33-42c0-a3c5-ade3831ab907" containerName="payload-pre-processing" containerID="cri-o://60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6" gracePeriod=25 Oct 30 18:55:46.652878 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:55:46.652835 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" podUID="577a654a-8c33-42c0-a3c5-ade3831ab907" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:55:56.653071 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:55:56.653037 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" podUID="577a654a-8c33-42c0-a3c5-ade3831ab907" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:55:57.784540 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:55:57.784498 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:56:02.436499 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.436473 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:56:02.592559 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.592528 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577a654a-8c33-42c0-a3c5-ade3831ab907-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577a654a-8c33-42c0-a3c5-ade3831ab907-plugins-config-volume\") pod \"577a654a-8c33-42c0-a3c5-ade3831ab907\" (UID: \"577a654a-8c33-42c0-a3c5-ade3831ab907\") " Oct 30 18:56:02.592747 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.592581 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577a654a-8c33-42c0-a3c5-ade3831ab907-kube-api-access-2fnj9\" (UniqueName: \"kubernetes.io/projected/577a654a-8c33-42c0-a3c5-ade3831ab907-kube-api-access-2fnj9\") pod \"577a654a-8c33-42c0-a3c5-ade3831ab907\" (UID: \"577a654a-8c33-42c0-a3c5-ade3831ab907\") " Oct 30 18:56:02.592992 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.592966 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577a654a-8c33-42c0-a3c5-ade3831ab907-plugins-config-volume" pod "577a654a-8c33-42c0-a3c5-ade3831ab907" (UID: "577a654a-8c33-42c0-a3c5-ade3831ab907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:02.593190 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.593173 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577a654a-8c33-42c0-a3c5-ade3831ab907-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:56:02.594927 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.594904 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577a654a-8c33-42c0-a3c5-ade3831ab907-kube-api-access-2fnj9" pod "577a654a-8c33-42c0-a3c5-ade3831ab907" (UID: "577a654a-8c33-42c0-a3c5-ade3831ab907"). InnerVolumeSpecName "kube-api-access-2fnj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:02.694109 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.694029 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fnj9\" (UniqueName: \"kubernetes.io/projected/577a654a-8c33-42c0-a3c5-ade3831ab907-kube-api-access-2fnj9\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:56:02.794203 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.794167 5959 generic.go:356] "Generic (PLEG): container finished" podID="577a654a-8c33-42c0-a3c5-ade3831ab907" containerID="60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6" exitCode=0 Oct 30 18:56:02.794383 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.794223 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" Oct 30 18:56:02.794383 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.794246 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" event={"ID":"577a654a-8c33-42c0-a3c5-ade3831ab907","Type":"ContainerDied","Data":"60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6"} Oct 30 18:56:02.794383 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.794313 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z" event={"ID":"577a654a-8c33-42c0-a3c5-ade3831ab907","Type":"ContainerDied","Data":"3f2d7435663fa78f14def2e1c51367ebbdb1e2388fd07d698cfeb6ae26ac86b1"} Oct 30 18:56:02.794383 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.794338 5959 scope.go:122] "RemoveContainer" containerID="60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6" Oct 30 18:56:02.820899 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.820769 5959 scope.go:122] "RemoveContainer" containerID="60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6" Oct 30 18:56:02.821189 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:56:02.821162 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6\": container with ID starting with 60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6 not found: ID does not exist" containerID="60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6" Oct 30 18:56:02.821277 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.821201 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6"} err="failed to get container status \"60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6\": rpc error: code = NotFound desc = could not find container \"60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6\": container with ID starting with 60ca409048b737c9baa25b2d0b90753148a312ed847c4eb80f929c6f4bac88c6 not found: ID does not exist" Oct 30 18:56:02.849670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.849634 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z"] Oct 30 18:56:02.853251 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:02.853228 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-da58b769-786c599f47-sf54z"] Oct 30 18:56:03.800536 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:03.800492 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577a654a-8c33-42c0-a3c5-ade3831ab907" path="/var/lib/kubelet/pods/577a654a-8c33-42c0-a3c5-ade3831ab907/volumes" Oct 30 18:56:11.836304 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836271 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr"] Oct 30 18:56:11.836917 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836903 5959 cpu_manager.go:414] "containerMap: removing container" podUID="462fad36-1de9-4523-90a8-2ef740e5f7af" containerName="payload-processing" Oct 30 18:56:11.836965 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836919 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462fad36-1de9-4523-90a8-2ef740e5f7af" containerName="payload-processing" Oct 30 18:56:11.836965 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836930 5959 cpu_manager.go:414] "containerMap: removing container" podUID="d114623c-5b11-4629-87c3-9ff8974cd0ad" containerName="payload-pre-processing" Oct 30 18:56:11.836965 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836936 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d114623c-5b11-4629-87c3-9ff8974cd0ad" containerName="payload-pre-processing" Oct 30 18:56:11.836965 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836950 5959 cpu_manager.go:414] "containerMap: removing container" podUID="daea56e1-2b7e-45dd-8554-727a7546d84c" containerName="payload-pre-processing" Oct 30 18:56:11.836965 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836956 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daea56e1-2b7e-45dd-8554-727a7546d84c" containerName="payload-pre-processing" Oct 30 18:56:11.836965 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836966 5959 cpu_manager.go:414] "containerMap: removing container" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" Oct 30 18:56:11.837138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836971 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" Oct 30 18:56:11.837138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.836999 5959 cpu_manager.go:414] "containerMap: removing container" podUID="577a654a-8c33-42c0-a3c5-ade3831ab907" containerName="payload-pre-processing" Oct 30 18:56:11.837138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837004 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577a654a-8c33-42c0-a3c5-ade3831ab907" containerName="payload-pre-processing" Oct 30 18:56:11.837138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837015 5959 cpu_manager.go:414] "containerMap: removing container" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" Oct 30 18:56:11.837138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837019 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" Oct 30 18:56:11.837138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837115 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="577a654a-8c33-42c0-a3c5-ade3831ab907" containerName="payload-pre-processing" Oct 30 18:56:11.837138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837124 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2a48a4-f066-4521-9150-8131ed67405d" containerName="istio-proxy" Oct 30 18:56:11.837138 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837136 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="462fad36-1de9-4523-90a8-2ef740e5f7af" containerName="payload-processing" Oct 30 18:56:11.837365 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837142 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d114623c-5b11-4629-87c3-9ff8974cd0ad" containerName="payload-pre-processing" Oct 30 18:56:11.837365 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837149 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="daea56e1-2b7e-45dd-8554-727a7546d84c" containerName="payload-pre-processing" Oct 30 18:56:11.837365 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.837156 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d700bcc8-04d2-45ef-b3c3-ddb9382d71a4" containerName="istio-proxy" Oct 30 18:56:11.842060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.842043 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:11.851348 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.851316 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8f1c37-serving-cert\"" Oct 30 18:56:11.856397 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.856372 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr"] Oct 30 18:56:11.943849 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.943804 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-maas-api-tls\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:11.944084 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.943998 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854j2\" (UniqueName: \"kubernetes.io/projected/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-kube-api-access-854j2\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:11.944240 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.944220 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:11.944315 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:11.944253 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-metrics-certs\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.047025 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.046996 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854j2\" (UniqueName: \"kubernetes.io/projected/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-kube-api-access-854j2\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.047275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.047258 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.047357 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.047295 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-metrics-certs\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.047414 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.047363 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-maas-api-tls\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.047959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.047933 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.055729 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.050274 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-metrics-certs\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.055729 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.051875 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-maas-api-tls\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.055729 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.055491 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854j2\" (UniqueName: \"kubernetes.io/projected/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-kube-api-access-854j2\") pod \"maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.158426 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.158397 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:12.495808 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.495775 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr"] Oct 30 18:56:12.498063 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:56:12.498033 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2de3a6d_9b71_4711_ae5d_36f2b4ecf00e.slice/crio-2b541a7e512b0ad3a50e7b25b1a39ba02fbbab30173e8fd9094c48f5969b00c5 WatchSource:0}: Error finding container 2b541a7e512b0ad3a50e7b25b1a39ba02fbbab30173e8fd9094c48f5969b00c5: Status 404 returned error can't find the container with id 2b541a7e512b0ad3a50e7b25b1a39ba02fbbab30173e8fd9094c48f5969b00c5 Oct 30 18:56:12.877149 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:12.877042 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" event={"ID":"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e","Type":"ContainerStarted","Data":"2b541a7e512b0ad3a50e7b25b1a39ba02fbbab30173e8fd9094c48f5969b00c5"} Oct 30 18:56:13.348968 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:13.348928 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:56:13.885153 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:13.885117 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" event={"ID":"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e","Type":"ContainerStarted","Data":"a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f"} Oct 30 18:56:13.885703 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:13.885318 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:13.905586 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:13.905537 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" podStartSLOduration=2.450303155 podStartE2EDuration="2.905520929s" totalImagesPullingTime="455.217774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:13.904090114 +0000 UTC m=+1154.232501496" watchObservedRunningTime="2026-10-30 18:56:13.905520929 +0000 UTC m=+1154.233932314" Oct 30 18:56:19.906021 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:19.905987 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:56:27.782619 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:56:27.782564 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:56:34.206303 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.206259 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5"] Oct 30 18:56:34.210801 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.210779 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:34.216705 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.216681 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-12a6e1\"" Oct 30 18:56:34.216856 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.216802 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-12a6e1-dockercfg-gsw65\"" Oct 30 18:56:34.218862 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.218837 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5"] Oct 30 18:56:34.274030 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.273994 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5db3987-807f-42a9-9c9f-ad5b7f41e393-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5\" (UID: \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:34.274415 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.274390 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhvd\" (UniqueName: \"kubernetes.io/projected/d5db3987-807f-42a9-9c9f-ad5b7f41e393-kube-api-access-2dhvd\") pod \"payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5\" (UID: \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:34.376221 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.376182 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhvd\" (UniqueName: \"kubernetes.io/projected/d5db3987-807f-42a9-9c9f-ad5b7f41e393-kube-api-access-2dhvd\") pod \"payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5\" (UID: \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:34.376438 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.376385 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5db3987-807f-42a9-9c9f-ad5b7f41e393-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5\" (UID: \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:34.377120 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.377096 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5db3987-807f-42a9-9c9f-ad5b7f41e393-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5\" (UID: \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:34.417676 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.411515 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhvd\" (UniqueName: \"kubernetes.io/projected/d5db3987-807f-42a9-9c9f-ad5b7f41e393-kube-api-access-2dhvd\") pod \"payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5\" (UID: \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:34.528583 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.528482 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:34.818260 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:34.818236 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5"] Oct 30 18:56:34.819030 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:56:34.819003 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5db3987_807f_42a9_9c9f_ad5b7f41e393.slice/crio-f0bd658999d40d4585fca71aaf20ae9fbc3161880a2f57b39fda33de6e66c48e WatchSource:0}: Error finding container f0bd658999d40d4585fca71aaf20ae9fbc3161880a2f57b39fda33de6e66c48e: Status 404 returned error can't find the container with id f0bd658999d40d4585fca71aaf20ae9fbc3161880a2f57b39fda33de6e66c48e Oct 30 18:56:35.073470 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:35.073368 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" event={"ID":"d5db3987-807f-42a9-9c9f-ad5b7f41e393","Type":"ContainerStarted","Data":"628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86"} Oct 30 18:56:35.073470 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:35.073429 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" event={"ID":"d5db3987-807f-42a9-9c9f-ad5b7f41e393","Type":"ContainerStarted","Data":"f0bd658999d40d4585fca71aaf20ae9fbc3161880a2f57b39fda33de6e66c48e"} Oct 30 18:56:35.073470 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:35.073453 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:35.092745 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:35.092655 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" podStartSLOduration=1.092639089 podStartE2EDuration="1.092639089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:35.091648424 +0000 UTC m=+1175.420059812" watchObservedRunningTime="2026-10-30 18:56:35.092639089 +0000 UTC m=+1175.421050586" Oct 30 18:56:36.396284 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.396249 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m"] Oct 30 18:56:36.401226 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.401205 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:36.409589 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.409559 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m"] Oct 30 18:56:36.411847 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.411828 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e7e190\"" Oct 30 18:56:36.412153 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.412115 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e7e190-dockercfg-hgvkn\"" Oct 30 18:56:36.442852 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.442808 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4"] Oct 30 18:56:36.448329 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.448285 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:36.457994 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.457619 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4"] Oct 30 18:56:36.512638 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.512294 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48810c6c-e579-4dc9-a5ed-1bb2d716f718-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4\" (UID: \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:36.512638 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.512447 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82df7f-7819-452a-a1f7-deb925eadfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m\" (UID: \"1b82df7f-7819-452a-a1f7-deb925eadfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:36.512638 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.512531 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqxj\" (UniqueName: \"kubernetes.io/projected/48810c6c-e579-4dc9-a5ed-1bb2d716f718-kube-api-access-tsqxj\") pod \"payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4\" (UID: \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:36.512939 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.512717 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67856\" (UniqueName: \"kubernetes.io/projected/1b82df7f-7819-452a-a1f7-deb925eadfce-kube-api-access-67856\") pod \"payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m\" (UID: \"1b82df7f-7819-452a-a1f7-deb925eadfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:36.622624 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.621698 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82df7f-7819-452a-a1f7-deb925eadfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m\" (UID: \"1b82df7f-7819-452a-a1f7-deb925eadfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:36.622624 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.621826 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqxj\" (UniqueName: \"kubernetes.io/projected/48810c6c-e579-4dc9-a5ed-1bb2d716f718-kube-api-access-tsqxj\") pod \"payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4\" (UID: \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:36.622624 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.622017 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67856\" (UniqueName: \"kubernetes.io/projected/1b82df7f-7819-452a-a1f7-deb925eadfce-kube-api-access-67856\") pod \"payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m\" (UID: \"1b82df7f-7819-452a-a1f7-deb925eadfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:36.622624 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.622132 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48810c6c-e579-4dc9-a5ed-1bb2d716f718-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4\" (UID: \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:36.628658 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.624746 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82df7f-7819-452a-a1f7-deb925eadfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m\" (UID: \"1b82df7f-7819-452a-a1f7-deb925eadfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:36.628658 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.625568 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48810c6c-e579-4dc9-a5ed-1bb2d716f718-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4\" (UID: \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:36.657955 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.640207 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67856\" (UniqueName: \"kubernetes.io/projected/1b82df7f-7819-452a-a1f7-deb925eadfce-kube-api-access-67856\") pod \"payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m\" (UID: \"1b82df7f-7819-452a-a1f7-deb925eadfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:36.661322 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.642272 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqxj\" (UniqueName: \"kubernetes.io/projected/48810c6c-e579-4dc9-a5ed-1bb2d716f718-kube-api-access-tsqxj\") pod \"payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4\" (UID: \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:36.718441 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.718400 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:36.767566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:36.767525 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:37.014471 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:37.014367 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m"] Oct 30 18:56:37.015885 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:56:37.015852 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b82df7f_7819_452a_a1f7_deb925eadfce.slice/crio-625f982ec5e65c98bd3f2c2b237428aa3c8998b16b1478d83feb8446e5809587 WatchSource:0}: Error finding container 625f982ec5e65c98bd3f2c2b237428aa3c8998b16b1478d83feb8446e5809587: Status 404 returned error can't find the container with id 625f982ec5e65c98bd3f2c2b237428aa3c8998b16b1478d83feb8446e5809587 Oct 30 18:56:37.090060 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:37.090034 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4"] Oct 30 18:56:37.091104 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:56:37.091079 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48810c6c_e579_4dc9_a5ed_1bb2d716f718.slice/crio-3e11ca82a90781d83cafa81c5fe9490db63e8f3f722f429907af35a5b78d0a53 WatchSource:0}: Error finding container 3e11ca82a90781d83cafa81c5fe9490db63e8f3f722f429907af35a5b78d0a53: Status 404 returned error can't find the container with id 3e11ca82a90781d83cafa81c5fe9490db63e8f3f722f429907af35a5b78d0a53 Oct 30 18:56:37.100399 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:37.100372 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" event={"ID":"48810c6c-e579-4dc9-a5ed-1bb2d716f718","Type":"ContainerStarted","Data":"3e11ca82a90781d83cafa81c5fe9490db63e8f3f722f429907af35a5b78d0a53"} Oct 30 18:56:37.102112 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:37.102088 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" event={"ID":"1b82df7f-7819-452a-a1f7-deb925eadfce","Type":"ContainerStarted","Data":"625f982ec5e65c98bd3f2c2b237428aa3c8998b16b1478d83feb8446e5809587"} Oct 30 18:56:37.102238 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:37.102218 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:37.119278 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:37.119225 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" podStartSLOduration=1.119210065 podStartE2EDuration="1.119210065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:37.117527815 +0000 UTC m=+1177.445939201" watchObservedRunningTime="2026-10-30 18:56:37.119210065 +0000 UTC m=+1177.447621443" Oct 30 18:56:38.109998 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:38.109964 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" event={"ID":"48810c6c-e579-4dc9-a5ed-1bb2d716f718","Type":"ContainerStarted","Data":"5dd5a861cb53b24cf7412bf809349e5436b28ef06633d6fcd43380d919be86c3"} Oct 30 18:56:38.110465 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:38.110039 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:38.111782 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:38.111754 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" event={"ID":"1b82df7f-7819-452a-a1f7-deb925eadfce","Type":"ContainerStarted","Data":"294e85dc5de1589f70b931e4e3f1536d56097d5ebfb53d857235c9c8c067680b"} Oct 30 18:56:38.129270 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:38.129204 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" podStartSLOduration=2.129183707 podStartE2EDuration="2.129183707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:38.126097105 +0000 UTC m=+1178.454508501" watchObservedRunningTime="2026-10-30 18:56:38.129183707 +0000 UTC m=+1178.457595093" Oct 30 18:56:44.917508 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:44.917427 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht"] Oct 30 18:56:44.923847 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:44.923820 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:44.933081 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:44.931149 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-12a6e1-serving-cert\"" Oct 30 18:56:44.939033 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:44.938991 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht"] Oct 30 18:56:45.061087 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.061037 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c42772-c97a-4909-b25a-ad4374473db0-openshift-service-ca\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.065212 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.065170 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbg2h\" (UniqueName: \"kubernetes.io/projected/43c42772-c97a-4909-b25a-ad4374473db0-kube-api-access-dbg2h\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.065871 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.065841 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-maas-api-tls\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.066183 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.066166 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-metrics-certs\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.168566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.168438 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c42772-c97a-4909-b25a-ad4374473db0-openshift-service-ca\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.168566 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.168546 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbg2h\" (UniqueName: \"kubernetes.io/projected/43c42772-c97a-4909-b25a-ad4374473db0-kube-api-access-dbg2h\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.168867 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.168782 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-maas-api-tls\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.168867 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.168843 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-metrics-certs\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.169171 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.169147 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c42772-c97a-4909-b25a-ad4374473db0-openshift-service-ca\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.172108 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.172079 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-maas-api-tls\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.172216 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.172117 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-metrics-certs\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.177506 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.177483 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbg2h\" (UniqueName: \"kubernetes.io/projected/43c42772-c97a-4909-b25a-ad4374473db0-kube-api-access-dbg2h\") pod \"maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.250209 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.250174 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:45.551507 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:45.551460 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht"] Oct 30 18:56:45.554693 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:56:45.554658 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c42772_c97a_4909_b25a_ad4374473db0.slice/crio-efc8521fad6de538072ade9ec0e4cd039e5db289749586def730752d49fc9ec2 WatchSource:0}: Error finding container efc8521fad6de538072ade9ec0e4cd039e5db289749586def730752d49fc9ec2: Status 404 returned error can't find the container with id efc8521fad6de538072ade9ec0e4cd039e5db289749586def730752d49fc9ec2 Oct 30 18:56:46.094883 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:46.094855 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:56:46.180066 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:46.180034 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" event={"ID":"43c42772-c97a-4909-b25a-ad4374473db0","Type":"ContainerStarted","Data":"e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338"} Oct 30 18:56:46.180255 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:46.180093 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" event={"ID":"43c42772-c97a-4909-b25a-ad4374473db0","Type":"ContainerStarted","Data":"efc8521fad6de538072ade9ec0e4cd039e5db289749586def730752d49fc9ec2"} Oct 30 18:56:46.180255 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:46.180139 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:46.203414 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:46.203357 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" podStartSLOduration=1.803889142 podStartE2EDuration="2.203340006s" totalImagesPullingTime="399.450864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:46.200066506 +0000 UTC m=+1186.528477893" watchObservedRunningTime="2026-10-30 18:56:46.203340006 +0000 UTC m=+1186.531751395" Oct 30 18:56:49.122346 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:49.122308 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:56:49.122797 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:49.122416 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:56:52.193436 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:52.193405 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:56:57.784654 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:56:57.784596 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:56:59.942521 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:59.942488 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4"] Oct 30 18:56:59.948527 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:56:59.948494 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m"] Oct 30 18:57:04.336909 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:04.336871 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:57:04.943009 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:04.942949 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" podUID="48810c6c-e579-4dc9-a5ed-1bb2d716f718" containerName="payload-processing" containerID="cri-o://5dd5a861cb53b24cf7412bf809349e5436b28ef06633d6fcd43380d919be86c3" gracePeriod=25 Oct 30 18:57:04.949572 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:04.949539 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" podUID="1b82df7f-7819-452a-a1f7-deb925eadfce" containerName="payload-pre-processing" containerID="cri-o://294e85dc5de1589f70b931e4e3f1536d56097d5ebfb53d857235c9c8c067680b" gracePeriod=25 Oct 30 18:57:05.372116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.368739 5959 generic.go:356] "Generic (PLEG): container finished" podID="48810c6c-e579-4dc9-a5ed-1bb2d716f718" containerID="5dd5a861cb53b24cf7412bf809349e5436b28ef06633d6fcd43380d919be86c3" exitCode=0 Oct 30 18:57:05.372116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.368852 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" event={"ID":"48810c6c-e579-4dc9-a5ed-1bb2d716f718","Type":"ContainerDied","Data":"5dd5a861cb53b24cf7412bf809349e5436b28ef06633d6fcd43380d919be86c3"} Oct 30 18:57:05.372116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.368906 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" event={"ID":"48810c6c-e579-4dc9-a5ed-1bb2d716f718","Type":"ContainerDied","Data":"3e11ca82a90781d83cafa81c5fe9490db63e8f3f722f429907af35a5b78d0a53"} Oct 30 18:57:05.372116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.368930 5959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e11ca82a90781d83cafa81c5fe9490db63e8f3f722f429907af35a5b78d0a53" Oct 30 18:57:05.372116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.371334 5959 generic.go:356] "Generic (PLEG): container finished" podID="1b82df7f-7819-452a-a1f7-deb925eadfce" containerID="294e85dc5de1589f70b931e4e3f1536d56097d5ebfb53d857235c9c8c067680b" exitCode=0 Oct 30 18:57:05.372116 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.371439 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" event={"ID":"1b82df7f-7819-452a-a1f7-deb925eadfce","Type":"ContainerDied","Data":"294e85dc5de1589f70b931e4e3f1536d56097d5ebfb53d857235c9c8c067680b"} Oct 30 18:57:05.376029 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.375901 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:57:05.390098 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.390067 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48810c6c-e579-4dc9-a5ed-1bb2d716f718-kube-api-access-tsqxj\" (UniqueName: \"kubernetes.io/projected/48810c6c-e579-4dc9-a5ed-1bb2d716f718-kube-api-access-tsqxj\") pod \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\" (UID: \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\") " Oct 30 18:57:05.390279 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.390259 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48810c6c-e579-4dc9-a5ed-1bb2d716f718-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48810c6c-e579-4dc9-a5ed-1bb2d716f718-plugins-config-volume\") pod \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\" (UID: \"48810c6c-e579-4dc9-a5ed-1bb2d716f718\") " Oct 30 18:57:05.391202 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.391063 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48810c6c-e579-4dc9-a5ed-1bb2d716f718-plugins-config-volume" pod "48810c6c-e579-4dc9-a5ed-1bb2d716f718" (UID: "48810c6c-e579-4dc9-a5ed-1bb2d716f718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:05.395277 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.393643 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48810c6c-e579-4dc9-a5ed-1bb2d716f718-kube-api-access-tsqxj" pod "48810c6c-e579-4dc9-a5ed-1bb2d716f718" (UID: "48810c6c-e579-4dc9-a5ed-1bb2d716f718"). InnerVolumeSpecName "kube-api-access-tsqxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:05.461855 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.461826 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:57:05.498513 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.498476 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqxj\" (UniqueName: \"kubernetes.io/projected/48810c6c-e579-4dc9-a5ed-1bb2d716f718-kube-api-access-tsqxj\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:05.499676 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.499651 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48810c6c-e579-4dc9-a5ed-1bb2d716f718-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:05.601027 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.600933 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b82df7f-7819-452a-a1f7-deb925eadfce-kube-api-access-67856\" (UniqueName: \"kubernetes.io/projected/1b82df7f-7819-452a-a1f7-deb925eadfce-kube-api-access-67856\") pod \"1b82df7f-7819-452a-a1f7-deb925eadfce\" (UID: \"1b82df7f-7819-452a-a1f7-deb925eadfce\") " Oct 30 18:57:05.601206 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.601136 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b82df7f-7819-452a-a1f7-deb925eadfce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82df7f-7819-452a-a1f7-deb925eadfce-plugins-config-volume\") pod \"1b82df7f-7819-452a-a1f7-deb925eadfce\" (UID: \"1b82df7f-7819-452a-a1f7-deb925eadfce\") " Oct 30 18:57:05.601484 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.601462 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b82df7f-7819-452a-a1f7-deb925eadfce-plugins-config-volume" pod "1b82df7f-7819-452a-a1f7-deb925eadfce" (UID: "1b82df7f-7819-452a-a1f7-deb925eadfce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:05.601688 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.601671 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b82df7f-7819-452a-a1f7-deb925eadfce-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:05.603367 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.603344 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b82df7f-7819-452a-a1f7-deb925eadfce-kube-api-access-67856" pod "1b82df7f-7819-452a-a1f7-deb925eadfce" (UID: "1b82df7f-7819-452a-a1f7-deb925eadfce"). InnerVolumeSpecName "kube-api-access-67856". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:05.703474 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:05.703411 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67856\" (UniqueName: \"kubernetes.io/projected/1b82df7f-7819-452a-a1f7-deb925eadfce-kube-api-access-67856\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:06.380875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:06.380778 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" event={"ID":"1b82df7f-7819-452a-a1f7-deb925eadfce","Type":"ContainerDied","Data":"625f982ec5e65c98bd3f2c2b237428aa3c8998b16b1478d83feb8446e5809587"} Oct 30 18:57:06.380875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:06.380865 5959 scope.go:122] "RemoveContainer" containerID="294e85dc5de1589f70b931e4e3f1536d56097d5ebfb53d857235c9c8c067680b" Oct 30 18:57:06.380875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:06.380872 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4" Oct 30 18:57:06.381485 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:06.380795 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m" Oct 30 18:57:06.403358 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:06.403327 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4"] Oct 30 18:57:06.409147 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:06.409119 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e7e190-758bf6f768-j56l4"] Oct 30 18:57:06.419227 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:06.419195 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m"] Oct 30 18:57:06.422904 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:06.422878 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e7e190-6b6ff8bc7f-z868m"] Oct 30 18:57:07.800199 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:07.800139 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b82df7f-7819-452a-a1f7-deb925eadfce" path="/var/lib/kubelet/pods/1b82df7f-7819-452a-a1f7-deb925eadfce/volumes" Oct 30 18:57:07.800799 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:07.800777 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48810c6c-e579-4dc9-a5ed-1bb2d716f718" path="/var/lib/kubelet/pods/48810c6c-e579-4dc9-a5ed-1bb2d716f718/volumes" Oct 30 18:57:25.593294 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:25.593258 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr"] Oct 30 18:57:25.594233 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:25.594172 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" podUID="d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" containerName="maas-api" containerID="cri-o://a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f" gracePeriod=30 Oct 30 18:57:26.123797 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.123766 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:57:26.279282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.276331 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-metrics-certs\") pod \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " Oct 30 18:57:26.279282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.276396 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-maas-api-tls\") pod \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " Oct 30 18:57:26.279282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.276584 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-kube-api-access-854j2\" (UniqueName: \"kubernetes.io/projected/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-kube-api-access-854j2\") pod \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " Oct 30 18:57:26.279282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.276750 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-openshift-service-ca\") pod \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\" (UID: \"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e\") " Oct 30 18:57:26.279282 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.278719 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-openshift-service-ca" pod "d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" (UID: "d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:26.280303 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.280271 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:26.281986 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.281955 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-kube-api-access-854j2" pod "d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" (UID: "d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e"). InnerVolumeSpecName "kube-api-access-854j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:26.283515 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.283373 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-metrics-certs" pod "d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" (UID: "d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:26.285004 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.284976 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-maas-api-tls" pod "d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" (UID: "d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:26.381450 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.381342 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:26.381450 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.381379 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:26.381450 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.381394 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854j2\" (UniqueName: \"kubernetes.io/projected/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e-kube-api-access-854j2\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:26.543426 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.543393 5959 generic.go:356] "Generic (PLEG): container finished" podID="d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" containerID="a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f" exitCode=0 Oct 30 18:57:26.543661 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.543479 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" Oct 30 18:57:26.543661 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.543479 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" event={"ID":"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e","Type":"ContainerDied","Data":"a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f"} Oct 30 18:57:26.543661 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.543581 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr" event={"ID":"d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e","Type":"ContainerDied","Data":"2b541a7e512b0ad3a50e7b25b1a39ba02fbbab30173e8fd9094c48f5969b00c5"} Oct 30 18:57:26.543661 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.543617 5959 scope.go:122] "RemoveContainer" containerID="a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f" Oct 30 18:57:26.555597 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.555574 5959 scope.go:122] "RemoveContainer" containerID="a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f" Oct 30 18:57:26.555914 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:57:26.555886 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f\": container with ID starting with a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f not found: ID does not exist" containerID="a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f" Oct 30 18:57:26.555969 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.555927 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f"} err="failed to get container status \"a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f\": rpc error: code = NotFound desc = could not find container \"a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f\": container with ID starting with a474cfbd44b209b79ff3cebc8c149145d20a401dc7ac6281fe682f2916b61e2f not found: ID does not exist" Oct 30 18:57:26.567899 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.567866 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr"] Oct 30 18:57:26.570457 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:26.570438 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1c37-6d4dd8f6c6-d9zmr"] Oct 30 18:57:27.786356 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:57:27.786315 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:57:27.800312 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:27.800267 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" path="/var/lib/kubelet/pods/d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e/volumes" Oct 30 18:57:29.536915 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.536882 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8"] Oct 30 18:57:29.537974 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.537953 5959 cpu_manager.go:414] "containerMap: removing container" podUID="d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" containerName="maas-api" Oct 30 18:57:29.538048 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.537978 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" containerName="maas-api" Oct 30 18:57:29.538048 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.538000 5959 cpu_manager.go:414] "containerMap: removing container" podUID="1b82df7f-7819-452a-a1f7-deb925eadfce" containerName="payload-pre-processing" Oct 30 18:57:29.538048 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.538009 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b82df7f-7819-452a-a1f7-deb925eadfce" containerName="payload-pre-processing" Oct 30 18:57:29.538048 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.538020 5959 cpu_manager.go:414] "containerMap: removing container" podUID="48810c6c-e579-4dc9-a5ed-1bb2d716f718" containerName="payload-processing" Oct 30 18:57:29.538048 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.538027 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48810c6c-e579-4dc9-a5ed-1bb2d716f718" containerName="payload-processing" Oct 30 18:57:29.538204 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.538192 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2de3a6d-9b71-4711-ae5d-36f2b4ecf00e" containerName="maas-api" Oct 30 18:57:29.538237 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.538205 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b82df7f-7819-452a-a1f7-deb925eadfce" containerName="payload-pre-processing" Oct 30 18:57:29.538237 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.538214 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="48810c6c-e579-4dc9-a5ed-1bb2d716f718" containerName="payload-processing" Oct 30 18:57:29.544062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.543835 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.548812 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.548783 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-54bc2b-serving-cert\"" Oct 30 18:57:29.554153 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.554122 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8"] Oct 30 18:57:29.628462 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.628427 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc726b7-5486-4b21-aaa7-d5be63d3c467-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.628682 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.628660 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6psv\" (UniqueName: \"kubernetes.io/projected/5bc726b7-5486-4b21-aaa7-d5be63d3c467-kube-api-access-d6psv\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.628743 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.628706 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-metrics-certs\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.628946 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.628919 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.730215 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.730178 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.730423 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.730276 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc726b7-5486-4b21-aaa7-d5be63d3c467-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.730423 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:57:29.730333 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-serving-cert: secret "maas-api-e2e-ipp-a-54bc2b-serving-cert" not found Oct 30 18:57:29.730423 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:57:29.730405 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls podName:5bc726b7-5486-4b21-aaa7-d5be63d3c467 nodeName:}" failed. No retries permitted until 2026-10-30 18:57:30.230385953 +0000 UTC m=+1230.558797316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls") pod "maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" (UID: "5bc726b7-5486-4b21-aaa7-d5be63d3c467") : secret "maas-api-e2e-ipp-a-54bc2b-serving-cert" not found Oct 30 18:57:29.730569 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.730472 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6psv\" (UniqueName: \"kubernetes.io/projected/5bc726b7-5486-4b21-aaa7-d5be63d3c467-kube-api-access-d6psv\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.730569 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.730508 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-metrics-certs\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.731058 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.731031 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc726b7-5486-4b21-aaa7-d5be63d3c467-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.733321 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.733300 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-metrics-certs\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:29.739788 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:29.739766 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6psv\" (UniqueName: \"kubernetes.io/projected/5bc726b7-5486-4b21-aaa7-d5be63d3c467-kube-api-access-d6psv\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:30.251460 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:30.251413 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:30.254469 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:30.254438 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:30.468128 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:30.468089 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:30.757688 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:30.757242 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8"] Oct 30 18:57:30.763106 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:57:30.763076 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc726b7_5486_4b21_aaa7_d5be63d3c467.slice/crio-ba18bd6d44a687f496c3d3d2f10d05afbc55243ffa7872b9f4d9bcf4320bd1d2 WatchSource:0}: Error finding container ba18bd6d44a687f496c3d3d2f10d05afbc55243ffa7872b9f4d9bcf4320bd1d2: Status 404 returned error can't find the container with id ba18bd6d44a687f496c3d3d2f10d05afbc55243ffa7872b9f4d9bcf4320bd1d2 Oct 30 18:57:31.264449 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:31.264414 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:57:31.614593 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:31.614501 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" event={"ID":"5bc726b7-5486-4b21-aaa7-d5be63d3c467","Type":"ContainerStarted","Data":"a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa"} Oct 30 18:57:31.614593 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:31.614556 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" event={"ID":"5bc726b7-5486-4b21-aaa7-d5be63d3c467","Type":"ContainerStarted","Data":"ba18bd6d44a687f496c3d3d2f10d05afbc55243ffa7872b9f4d9bcf4320bd1d2"} Oct 30 18:57:31.614810 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:31.614713 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:31.637636 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:31.637551 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" podStartSLOduration=2.062244599 podStartE2EDuration="2.637535629s" totalImagesPullingTime="575.29103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:31.635439919 +0000 UTC m=+1231.963851305" watchObservedRunningTime="2026-10-30 18:57:31.637535629 +0000 UTC m=+1231.965947051" Oct 30 18:57:37.631906 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:37.631876 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:57:44.235375 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:44.235340 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5"] Oct 30 18:57:49.236074 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.235999 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" podUID="d5db3987-807f-42a9-9c9f-ad5b7f41e393" containerName="payload-pre-processing" containerID="cri-o://628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86" gracePeriod=25 Oct 30 18:57:49.531643 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.531573 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:57:49.603827 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.603792 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5db3987-807f-42a9-9c9f-ad5b7f41e393-kube-api-access-2dhvd\" (UniqueName: \"kubernetes.io/projected/d5db3987-807f-42a9-9c9f-ad5b7f41e393-kube-api-access-2dhvd\") pod \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\" (UID: \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\") " Oct 30 18:57:49.604044 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.603962 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5db3987-807f-42a9-9c9f-ad5b7f41e393-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5db3987-807f-42a9-9c9f-ad5b7f41e393-plugins-config-volume\") pod \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\" (UID: \"d5db3987-807f-42a9-9c9f-ad5b7f41e393\") " Oct 30 18:57:49.604389 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.604361 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5db3987-807f-42a9-9c9f-ad5b7f41e393-plugins-config-volume" pod "d5db3987-807f-42a9-9c9f-ad5b7f41e393" (UID: "d5db3987-807f-42a9-9c9f-ad5b7f41e393"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:49.604516 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.604493 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5db3987-807f-42a9-9c9f-ad5b7f41e393-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:49.606273 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.606247 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5db3987-807f-42a9-9c9f-ad5b7f41e393-kube-api-access-2dhvd" pod "d5db3987-807f-42a9-9c9f-ad5b7f41e393" (UID: "d5db3987-807f-42a9-9c9f-ad5b7f41e393"). InnerVolumeSpecName "kube-api-access-2dhvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:49.706510 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.706466 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhvd\" (UniqueName: \"kubernetes.io/projected/d5db3987-807f-42a9-9c9f-ad5b7f41e393-kube-api-access-2dhvd\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:49.778580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.778492 5959 generic.go:356] "Generic (PLEG): container finished" podID="d5db3987-807f-42a9-9c9f-ad5b7f41e393" containerID="628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86" exitCode=0 Oct 30 18:57:49.778580 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.778563 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" Oct 30 18:57:49.778798 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.778595 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" event={"ID":"d5db3987-807f-42a9-9c9f-ad5b7f41e393","Type":"ContainerDied","Data":"628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86"} Oct 30 18:57:49.778798 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.778696 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5" event={"ID":"d5db3987-807f-42a9-9c9f-ad5b7f41e393","Type":"ContainerDied","Data":"f0bd658999d40d4585fca71aaf20ae9fbc3161880a2f57b39fda33de6e66c48e"} Oct 30 18:57:49.778798 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.778732 5959 scope.go:122] "RemoveContainer" containerID="628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86" Oct 30 18:57:49.789842 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.789821 5959 scope.go:122] "RemoveContainer" containerID="628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86" Oct 30 18:57:49.793874 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:57:49.793850 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86\": container with ID starting with 628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86 not found: ID does not exist" containerID="628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86" Oct 30 18:57:49.794022 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.793884 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86"} err="failed to get container status \"628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86\": rpc error: code = NotFound desc = could not find container \"628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86\": container with ID starting with 628bc07fff54c3e9a2ac57c0333a779a3a280d920763d898aeda00ffd53dfa86 not found: ID does not exist" Oct 30 18:57:49.810592 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.810559 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5"] Oct 30 18:57:49.815792 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:49.815766 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-12a6e1-858d56dc8c-m8kb5"] Oct 30 18:57:51.798107 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:57:51.798077 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5db3987-807f-42a9-9c9f-ad5b7f41e393" path="/var/lib/kubelet/pods/d5db3987-807f-42a9-9c9f-ad5b7f41e393/volumes" Oct 30 18:57:57.781177 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:57:57.781136 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:58:02.963239 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:02.963200 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht"] Oct 30 18:58:02.964148 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:02.964102 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" podUID="43c42772-c97a-4909-b25a-ad4374473db0" containerName="maas-api" containerID="cri-o://e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338" gracePeriod=30 Oct 30 18:58:03.272910 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.272883 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:58:03.316140 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.316110 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c42772-c97a-4909-b25a-ad4374473db0-kube-api-access-dbg2h\" (UniqueName: \"kubernetes.io/projected/43c42772-c97a-4909-b25a-ad4374473db0-kube-api-access-dbg2h\") pod \"43c42772-c97a-4909-b25a-ad4374473db0\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " Oct 30 18:58:03.316502 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.316475 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c42772-c97a-4909-b25a-ad4374473db0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c42772-c97a-4909-b25a-ad4374473db0-openshift-service-ca\") pod \"43c42772-c97a-4909-b25a-ad4374473db0\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " Oct 30 18:58:03.316738 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.316708 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-metrics-certs\") pod \"43c42772-c97a-4909-b25a-ad4374473db0\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " Oct 30 18:58:03.316967 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.316943 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-maas-api-tls\") pod \"43c42772-c97a-4909-b25a-ad4374473db0\" (UID: \"43c42772-c97a-4909-b25a-ad4374473db0\") " Oct 30 18:58:03.317983 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.317956 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c42772-c97a-4909-b25a-ad4374473db0-openshift-service-ca" pod "43c42772-c97a-4909-b25a-ad4374473db0" (UID: "43c42772-c97a-4909-b25a-ad4374473db0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:03.334062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.333842 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-metrics-certs" pod "43c42772-c97a-4909-b25a-ad4374473db0" (UID: "43c42772-c97a-4909-b25a-ad4374473db0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:03.334062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.333934 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-maas-api-tls" pod "43c42772-c97a-4909-b25a-ad4374473db0" (UID: "43c42772-c97a-4909-b25a-ad4374473db0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:03.334062 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.334030 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c42772-c97a-4909-b25a-ad4374473db0-kube-api-access-dbg2h" pod "43c42772-c97a-4909-b25a-ad4374473db0" (UID: "43c42772-c97a-4909-b25a-ad4374473db0"). InnerVolumeSpecName "kube-api-access-dbg2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:03.440256 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.420030 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c42772-c97a-4909-b25a-ad4374473db0-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:03.440539 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.440515 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:03.440659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.440644 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c42772-c97a-4909-b25a-ad4374473db0-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:03.440751 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.440737 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbg2h\" (UniqueName: \"kubernetes.io/projected/43c42772-c97a-4909-b25a-ad4374473db0-kube-api-access-dbg2h\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:03.879461 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.879425 5959 generic.go:356] "Generic (PLEG): container finished" podID="43c42772-c97a-4909-b25a-ad4374473db0" containerID="e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338" exitCode=0 Oct 30 18:58:03.879659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.879468 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" event={"ID":"43c42772-c97a-4909-b25a-ad4374473db0","Type":"ContainerDied","Data":"e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338"} Oct 30 18:58:03.879659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.879534 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" event={"ID":"43c42772-c97a-4909-b25a-ad4374473db0","Type":"ContainerDied","Data":"efc8521fad6de538072ade9ec0e4cd039e5db289749586def730752d49fc9ec2"} Oct 30 18:58:03.879659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.879558 5959 scope.go:122] "RemoveContainer" containerID="e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338" Oct 30 18:58:03.879659 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.879477 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht" Oct 30 18:58:03.889889 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.889873 5959 scope.go:122] "RemoveContainer" containerID="e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338" Oct 30 18:58:03.890224 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:58:03.890201 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338\": container with ID starting with e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338 not found: ID does not exist" containerID="e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338" Oct 30 18:58:03.890312 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.890231 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338"} err="failed to get container status \"e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338\": rpc error: code = NotFound desc = could not find container \"e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338\": container with ID starting with e19e9a2ee5e75a2b7b2ec50ac617c23f046ce5bdd10c8ae09020f4ab835a9338 not found: ID does not exist" Oct 30 18:58:03.896548 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.896525 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht"] Oct 30 18:58:03.910026 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:03.909997 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-12a6e1-b98577db5-gs9ht"] Oct 30 18:58:05.798883 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:05.798839 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c42772-c97a-4909-b25a-ad4374473db0" path="/var/lib/kubelet/pods/43c42772-c97a-4909-b25a-ad4374473db0/volumes" Oct 30 18:58:10.896858 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:10.896779 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:58:14.752487 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:14.752446 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg"] Oct 30 18:58:14.753024 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:14.752735 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" containerID="cri-o://2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372" gracePeriod=30 Oct 30 18:58:19.633462 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.633423 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z"] Oct 30 18:58:19.634108 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.634092 5959 cpu_manager.go:414] "containerMap: removing container" podUID="43c42772-c97a-4909-b25a-ad4374473db0" containerName="maas-api" Oct 30 18:58:19.634155 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.634110 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c42772-c97a-4909-b25a-ad4374473db0" containerName="maas-api" Oct 30 18:58:19.634155 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.634138 5959 cpu_manager.go:414] "containerMap: removing container" podUID="d5db3987-807f-42a9-9c9f-ad5b7f41e393" containerName="payload-pre-processing" Oct 30 18:58:19.634155 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.634144 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5db3987-807f-42a9-9c9f-ad5b7f41e393" containerName="payload-pre-processing" Oct 30 18:58:19.634250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.634230 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c42772-c97a-4909-b25a-ad4374473db0" containerName="maas-api" Oct 30 18:58:19.634250 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.634242 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5db3987-807f-42a9-9c9f-ad5b7f41e393" containerName="payload-pre-processing" Oct 30 18:58:19.637742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.637726 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:19.647644 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.645024 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f778f5-dockercfg-b2dw7\"" Oct 30 18:58:19.647644 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.645108 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f778f5\"" Oct 30 18:58:19.648170 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.647813 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z"] Oct 30 18:58:19.705640 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.705594 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmz2h\" (UniqueName: \"kubernetes.io/projected/dd74c92b-bc79-431c-a713-7135937db8ff-kube-api-access-cmz2h\") pod \"payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z\" (UID: \"dd74c92b-bc79-431c-a713-7135937db8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:19.705912 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.705889 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd74c92b-bc79-431c-a713-7135937db8ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z\" (UID: \"dd74c92b-bc79-431c-a713-7135937db8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:19.810342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.808461 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd74c92b-bc79-431c-a713-7135937db8ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z\" (UID: \"dd74c92b-bc79-431c-a713-7135937db8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:19.810342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.808779 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmz2h\" (UniqueName: \"kubernetes.io/projected/dd74c92b-bc79-431c-a713-7135937db8ff-kube-api-access-cmz2h\") pod \"payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z\" (UID: \"dd74c92b-bc79-431c-a713-7135937db8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:19.810342 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.810071 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd74c92b-bc79-431c-a713-7135937db8ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z\" (UID: \"dd74c92b-bc79-431c-a713-7135937db8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:19.829956 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.829925 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmz2h\" (UniqueName: \"kubernetes.io/projected/dd74c92b-bc79-431c-a713-7135937db8ff-kube-api-access-cmz2h\") pod \"payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z\" (UID: \"dd74c92b-bc79-431c-a713-7135937db8ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:19.953054 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:19.953021 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:20.013513 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.013484 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:58:20.115212 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115177 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-envoy\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.115387 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115224 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istiod-ca-cert\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.115387 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115325 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-kube-api-access-dpgg4\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-kube-api-access-dpgg4\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.115524 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115430 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-podinfo\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.115524 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115494 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-credential-socket\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.115664 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115645 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-token\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-token\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.115798 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115740 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-credential-socket" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:20.115798 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115747 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istiod-ca-cert" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:20.115798 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115783 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-certs\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.116276 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115861 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-data\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.116276 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.115896 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-socket\") pod \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\" (UID: \"3b59a8b4-a585-47b3-bc59-a48c4cce09e9\") " Oct 30 18:58:20.116276 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.116063 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-certs" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:20.116276 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.116184 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-data" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:20.116505 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.116328 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-socket" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:20.116729 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.116704 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.116837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.116735 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.116837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.116751 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.116837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.116765 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.116837 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.116779 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.118530 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.118500 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-token" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:20.118530 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.118521 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-kube-api-access-dpgg4" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "kube-api-access-dpgg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:20.118756 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.118733 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-envoy" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:20.118977 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.118959 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-podinfo" pod "3b59a8b4-a585-47b3-bc59-a48c4cce09e9" (UID: "3b59a8b4-a585-47b3-bc59-a48c4cce09e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:58:20.201716 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.201619 5959 generic.go:356] "Generic (PLEG): container finished" podID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerID="2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372" exitCode=0 Oct 30 18:58:20.201892 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.201778 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" event={"ID":"3b59a8b4-a585-47b3-bc59-a48c4cce09e9","Type":"ContainerDied","Data":"2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372"} Oct 30 18:58:20.201972 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.201904 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" event={"ID":"3b59a8b4-a585-47b3-bc59-a48c4cce09e9","Type":"ContainerDied","Data":"6cb992e8058a859423b166c56094d6c9fbaccb36da0c9713070b3e3418cb79fe"} Oct 30 18:58:20.201972 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.201949 5959 scope.go:122] "RemoveContainer" containerID="2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372" Oct 30 18:58:20.202899 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.202161 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg" Oct 30 18:58:20.214059 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.214033 5959 scope.go:122] "RemoveContainer" containerID="2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372" Oct 30 18:58:20.214344 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:58:20.214325 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372\": container with ID starting with 2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372 not found: ID does not exist" containerID="2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372" Oct 30 18:58:20.214414 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.214352 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372"} err="failed to get container status \"2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372\": rpc error: code = NotFound desc = could not find container \"2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372\": container with ID starting with 2fa2509ce7397416b1c71e25cc8d369e5a097f882d9775357343573697498372 not found: ID does not exist" Oct 30 18:58:20.217730 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.217710 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.217730 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.217731 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.217845 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.217742 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpgg4\" (UniqueName: \"kubernetes.io/projected/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-kube-api-access-dpgg4\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.217845 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.217751 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b59a8b4-a585-47b3-bc59-a48c4cce09e9-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:20.228542 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.228519 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg"] Oct 30 18:58:20.237378 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.237354 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-b7vtg"] Oct 30 18:58:20.258620 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:20.258582 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z"] Oct 30 18:58:20.260248 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:58:20.260222 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd74c92b_bc79_431c_a713_7135937db8ff.slice/crio-3283fc97c5a1e20a7ee43337bc0b52dd644efb4144f2bfb9abc0cd07dff93066 WatchSource:0}: Error finding container 3283fc97c5a1e20a7ee43337bc0b52dd644efb4144f2bfb9abc0cd07dff93066: Status 404 returned error can't find the container with id 3283fc97c5a1e20a7ee43337bc0b52dd644efb4144f2bfb9abc0cd07dff93066 Oct 30 18:58:21.212643 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:21.212154 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" event={"ID":"dd74c92b-bc79-431c-a713-7135937db8ff","Type":"ContainerStarted","Data":"289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741"} Oct 30 18:58:21.212643 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:21.212323 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:21.212643 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:21.212402 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" event={"ID":"dd74c92b-bc79-431c-a713-7135937db8ff","Type":"ContainerStarted","Data":"3283fc97c5a1e20a7ee43337bc0b52dd644efb4144f2bfb9abc0cd07dff93066"} Oct 30 18:58:21.234838 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:21.234787 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" podStartSLOduration=2.234771114 podStartE2EDuration="2.234771114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:21.233406435 +0000 UTC m=+1281.561817822" watchObservedRunningTime="2026-10-30 18:58:21.234771114 +0000 UTC m=+1281.563182498" Oct 30 18:58:21.797830 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:21.797794 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" path="/var/lib/kubelet/pods/3b59a8b4-a585-47b3-bc59-a48c4cce09e9/volumes" Oct 30 18:58:23.766467 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.766428 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5"] Oct 30 18:58:23.767170 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.767152 5959 cpu_manager.go:414] "containerMap: removing container" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" Oct 30 18:58:23.767170 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.767171 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" Oct 30 18:58:23.767321 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.767288 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b59a8b4-a585-47b3-bc59-a48c4cce09e9" containerName="istio-proxy" Oct 30 18:58:23.771484 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.771465 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.778283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.778252 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5"] Oct 30 18:58:23.781648 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.781313 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-fe0612-serving-cert\"" Oct 30 18:58:23.874073 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.874035 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b482d51-e94c-4e1e-8082-35e825cce973-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.874280 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.874257 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-metrics-certs\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.874648 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.874551 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.874780 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.874693 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cst8t\" (UniqueName: \"kubernetes.io/projected/0b482d51-e94c-4e1e-8082-35e825cce973-kube-api-access-cst8t\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.975618 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.975584 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b482d51-e94c-4e1e-8082-35e825cce973-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.975811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.975710 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-metrics-certs\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.975811 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.975774 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.975896 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.975813 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cst8t\" (UniqueName: \"kubernetes.io/projected/0b482d51-e94c-4e1e-8082-35e825cce973-kube-api-access-cst8t\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.976382 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.976344 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b482d51-e94c-4e1e-8082-35e825cce973-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.978591 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.978568 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-metrics-certs\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.978698 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.978568 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:23.984622 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:23.984585 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cst8t\" (UniqueName: \"kubernetes.io/projected/0b482d51-e94c-4e1e-8082-35e825cce973-kube-api-access-cst8t\") pod \"maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:24.088916 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:24.088823 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:24.426979 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:24.426947 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5"] Oct 30 18:58:24.437667 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:58:24.435209 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b482d51_e94c_4e1e_8082_35e825cce973.slice/crio-4a398dbce7983caffdecc00e28c3901705f0e1d4d93926095eb806ca4cad9a4e WatchSource:0}: Error finding container 4a398dbce7983caffdecc00e28c3901705f0e1d4d93926095eb806ca4cad9a4e: Status 404 returned error can't find the container with id 4a398dbce7983caffdecc00e28c3901705f0e1d4d93926095eb806ca4cad9a4e Oct 30 18:58:25.260749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.260710 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" event={"ID":"0b482d51-e94c-4e1e-8082-35e825cce973","Type":"ContainerStarted","Data":"c1b1f411be054401b76b5478dbda83d836d8b82972b87d0bd749ee4255e8e39f"} Oct 30 18:58:25.261160 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.260800 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" event={"ID":"0b482d51-e94c-4e1e-8082-35e825cce973","Type":"ContainerStarted","Data":"4a398dbce7983caffdecc00e28c3901705f0e1d4d93926095eb806ca4cad9a4e"} Oct 30 18:58:25.261160 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.260868 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:25.280059 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.280008 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" podStartSLOduration=1.843251681 podStartE2EDuration="2.279994213s" totalImagesPullingTime="436.742532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:25.276886006 +0000 UTC m=+1285.605297389" watchObservedRunningTime="2026-10-30 18:58:25.279994213 +0000 UTC m=+1285.608405597" Oct 30 18:58:25.650929 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.650891 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf"] Oct 30 18:58:25.656414 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.656386 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.671283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.670743 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-655193-serving-cert\"" Oct 30 18:58:25.671283 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.670789 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf"] Oct 30 18:58:25.814000 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.813954 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.814291 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.814115 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvlg5\" (UniqueName: \"kubernetes.io/projected/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-kube-api-access-hvlg5\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.814291 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.814178 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.814477 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.814390 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-metrics-certs\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.915912 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.915836 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.916073 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.916013 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-metrics-certs\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.916134 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.916085 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.916293 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.916248 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvlg5\" (UniqueName: \"kubernetes.io/projected/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-kube-api-access-hvlg5\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.916851 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.916828 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.918406 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.918385 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.918574 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.918560 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-metrics-certs\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.924736 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.924706 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvlg5\" (UniqueName: \"kubernetes.io/projected/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-kube-api-access-hvlg5\") pod \"maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:25.979994 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:25.979963 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:26.299762 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:26.281393 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf"] Oct 30 18:58:27.308263 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:27.308233 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" event={"ID":"c823ebca-d0aa-4592-8d6e-a5273a34f3b3","Type":"ContainerStarted","Data":"0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854"} Oct 30 18:58:27.308754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:27.308279 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" event={"ID":"c823ebca-d0aa-4592-8d6e-a5273a34f3b3","Type":"ContainerStarted","Data":"1059b51725972db7da8f4e9d0c2fd4f5d4e77532c8abdf6382d02236597dcfd8"} Oct 30 18:58:27.308754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:27.308452 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:27.326290 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:27.326241 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" podStartSLOduration=1.91641733 podStartE2EDuration="2.326224997s" totalImagesPullingTime="409.807667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:27.325374599 +0000 UTC m=+1287.653785985" watchObservedRunningTime="2026-10-30 18:58:27.326224997 +0000 UTC m=+1287.654636381" Oct 30 18:58:27.781046 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:58:27.781008 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:58:28.332311 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.332276 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs"] Oct 30 18:58:28.337834 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.337809 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.347175 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.346811 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs"] Oct 30 18:58:28.348413 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.348386 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a014d1-serving-cert\"" Oct 30 18:58:28.454587 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.454547 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-metrics-certs\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.454894 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.454765 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxc5\" (UniqueName: \"kubernetes.io/projected/f634b8e0-7f6c-491d-803a-ada7f7b0627f-kube-api-access-wjxc5\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.454894 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.454832 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f634b8e0-7f6c-491d-803a-ada7f7b0627f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.455023 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.454964 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.556360 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.556317 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-metrics-certs\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.556555 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.556443 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxc5\" (UniqueName: \"kubernetes.io/projected/f634b8e0-7f6c-491d-803a-ada7f7b0627f-kube-api-access-wjxc5\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.556555 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.556474 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f634b8e0-7f6c-491d-803a-ada7f7b0627f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.556555 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.556530 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.557245 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.557165 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f634b8e0-7f6c-491d-803a-ada7f7b0627f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.558938 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.558920 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-metrics-certs\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.559080 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.559059 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.564893 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.564873 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxc5\" (UniqueName: \"kubernetes.io/projected/f634b8e0-7f6c-491d-803a-ada7f7b0627f-kube-api-access-wjxc5\") pod \"maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.661864 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.661828 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:28.963071 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:28.962918 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs"] Oct 30 18:58:28.967250 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:58:28.966823 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf634b8e0_7f6c_491d_803a_ada7f7b0627f.slice/crio-962ef2cb63a1b7e69737b6d20dd8f43aefc292925a9b9352c58f81ee9a9cc1ab WatchSource:0}: Error finding container 962ef2cb63a1b7e69737b6d20dd8f43aefc292925a9b9352c58f81ee9a9cc1ab: Status 404 returned error can't find the container with id 962ef2cb63a1b7e69737b6d20dd8f43aefc292925a9b9352c58f81ee9a9cc1ab Oct 30 18:58:29.325041 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:29.324950 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" event={"ID":"f634b8e0-7f6c-491d-803a-ada7f7b0627f","Type":"ContainerStarted","Data":"962ef2cb63a1b7e69737b6d20dd8f43aefc292925a9b9352c58f81ee9a9cc1ab"} Oct 30 18:58:30.337148 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:30.337114 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" event={"ID":"f634b8e0-7f6c-491d-803a-ada7f7b0627f","Type":"ContainerStarted","Data":"5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623"} Oct 30 18:58:30.337562 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:30.337166 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:30.356345 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:30.356297 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" podStartSLOduration=1.935457564 podStartE2EDuration="2.356283333s" totalImagesPullingTime="420.825769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:30.354539868 +0000 UTC m=+1290.682951265" watchObservedRunningTime="2026-10-30 18:58:30.356283333 +0000 UTC m=+1290.684694717" Oct 30 18:58:31.313201 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:31.313171 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:58:32.223344 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:32.223315 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:58:33.320570 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:33.320544 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:58:36.353090 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:36.353061 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 18:58:41.956388 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:41.956356 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:58:44.109633 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:44.105995 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp"] Oct 30 18:58:44.112066 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:44.112028 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" containerID="cri-o://08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab" gracePeriod=30 Oct 30 18:58:49.423380 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.423352 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:58:49.490912 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.490869 5959 generic.go:356] "Generic (PLEG): container finished" podID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerID="08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab" exitCode=0 Oct 30 18:58:49.491109 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.490935 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" Oct 30 18:58:49.491109 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.490948 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" event={"ID":"2f514dd9-e3d4-4533-b6ba-422964e0ae43","Type":"ContainerDied","Data":"08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab"} Oct 30 18:58:49.491109 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.491011 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp" event={"ID":"2f514dd9-e3d4-4533-b6ba-422964e0ae43","Type":"ContainerDied","Data":"9af1ccd3e5a5475a16f5abdda3b3e328eaf34c31a36c955c2dfb028f1a37988d"} Oct 30 18:58:49.491109 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.491040 5959 scope.go:122] "RemoveContainer" containerID="08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab" Oct 30 18:58:49.503398 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.503365 5959 scope.go:122] "RemoveContainer" containerID="08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab" Oct 30 18:58:49.503691 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:58:49.503674 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab\": container with ID starting with 08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab not found: ID does not exist" containerID="08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab" Oct 30 18:58:49.503754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.503700 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab"} err="failed to get container status \"08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab\": rpc error: code = NotFound desc = could not find container \"08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab\": container with ID starting with 08d697fb62bcec06cf1e3191be38020cdb963453b92f392306792861055957ab not found: ID does not exist" Oct 30 18:58:49.521852 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.521823 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-kube-api-access-wb2qj\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-kube-api-access-wb2qj\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522017 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.521999 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-token\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-token\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522126 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522109 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-socket\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522196 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522166 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-data\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522261 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522237 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-credential-socket\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522321 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522294 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-podinfo\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522374 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522335 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istiod-ca-cert\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522374 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522367 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-envoy\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522489 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522410 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-socket" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.522489 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522420 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-certs\") pod \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\" (UID: \"2f514dd9-e3d4-4533-b6ba-422964e0ae43\") " Oct 30 18:58:49.522489 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522421 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-data" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.522780 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522740 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istiod-ca-cert" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.522896 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.522861 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-credential-socket" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.523200 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.523180 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.523275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.523206 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.523275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.523220 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.523275 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.523234 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.523391 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.523323 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-certs" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.524243 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.524223 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-kube-api-access-wb2qj" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "kube-api-access-wb2qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.524849 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.524826 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-envoy" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.525144 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.525129 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-token" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.525144 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.525135 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-podinfo" pod "2f514dd9-e3d4-4533-b6ba-422964e0ae43" (UID: "2f514dd9-e3d4-4533-b6ba-422964e0ae43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:58:49.624455 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.624424 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.624455 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.624452 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.624455 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.624462 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.624757 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.624471 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f514dd9-e3d4-4533-b6ba-422964e0ae43-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.624757 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.624481 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb2qj\" (UniqueName: \"kubernetes.io/projected/2f514dd9-e3d4-4533-b6ba-422964e0ae43-kube-api-access-wb2qj\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:58:49.831423 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.831391 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp"] Oct 30 18:58:49.835292 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:49.835248 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-rjtfp"] Oct 30 18:58:51.797473 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:51.797438 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" path="/var/lib/kubelet/pods/2f514dd9-e3d4-4533-b6ba-422964e0ae43/volumes" Oct 30 18:58:52.283521 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.283481 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4"] Oct 30 18:58:52.284124 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.284107 5959 cpu_manager.go:414] "containerMap: removing container" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" Oct 30 18:58:52.284124 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.284123 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" Oct 30 18:58:52.284289 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.284277 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f514dd9-e3d4-4533-b6ba-422964e0ae43" containerName="istio-proxy" Oct 30 18:58:52.289737 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.289707 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.308151 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.307620 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4"] Oct 30 18:58:52.368955 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.368917 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-token\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.369122 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.368963 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-data\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.369122 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.369104 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-certs\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.369196 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.369136 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-podinfo\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.369196 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.369184 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-envoy\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.369370 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.369349 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-credential-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.369444 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.369385 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdrj\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-kube-api-access-gmdrj\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.369444 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.369419 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.369550 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.369512 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istiod-ca-cert\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.386462 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.386433 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fe0612-openshift-default-dockercfg-d4m8b\"" Oct 30 18:58:52.471198 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471159 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-token\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471384 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471225 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-data\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471384 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471358 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-certs\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471484 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471391 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-podinfo\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471484 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471430 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-envoy\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471594 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471550 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-credential-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471594 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471573 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdrj\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-kube-api-access-gmdrj\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471720 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471640 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471720 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471647 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-data\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471720 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471684 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istiod-ca-cert\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471875 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471748 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-certs\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.471942 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.471894 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.472096 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.472078 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-credential-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.472399 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.472360 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istiod-ca-cert\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.473984 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.473962 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-envoy\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.474089 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.474079 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-podinfo\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.479662 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.479638 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-token\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.480178 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.480157 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdrj\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-kube-api-access-gmdrj\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:52.616695 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:52.616591 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:53.014623 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:53.012937 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4"] Oct 30 18:58:53.032406 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:53.029126 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:58:53.032406 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:53.029216 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:58:53.032406 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:53.029259 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 18:58:53.533013 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:53.532978 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" event={"ID":"ccd42b08-75db-4201-8f3c-6c21fddf22eb","Type":"ContainerStarted","Data":"7c3a92b5a20cd532e447b5ca32384f46d92bfeccfb6d3e86aa59732fd49a4198"} Oct 30 18:58:53.533229 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:53.533032 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" event={"ID":"ccd42b08-75db-4201-8f3c-6c21fddf22eb","Type":"ContainerStarted","Data":"e19895f9bd11538b487b6f862d68fb5d098d0de5faa5aef14c1926e9d3b13ad2"} Oct 30 18:58:53.553364 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:53.553309 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" podStartSLOduration=1.553294379 podStartE2EDuration="1.553294379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:53.552212092 +0000 UTC m=+1313.880623476" watchObservedRunningTime="2026-10-30 18:58:53.553294379 +0000 UTC m=+1313.881705765" Oct 30 18:58:53.617090 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:53.617061 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:54.622761 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:54.622712 5959 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:58:54.623200 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:54.622785 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:58:55.620631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:55.620154 5959 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:58:55.620631 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:55.620219 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:58:56.625666 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:56.620976 5959 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:58:56.625666 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:56.621040 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:58:57.622054 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:57.621939 5959 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:58:57.622054 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:57.622002 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:58:57.782948 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:58:57.782904 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:58:58.618628 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:58.618154 5959 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:58:58.618628 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:58.618235 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:58:59.622238 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:59.622208 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:59.622660 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:59.622636 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:58:59.623280 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:58:59.623260 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:59:14.281778 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:14.281739 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:59:19.955320 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:19.955282 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8"] Oct 30 18:59:19.955879 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:19.955556 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" podUID="5bc726b7-5486-4b21-aaa7-d5be63d3c467" containerName="maas-api" containerID="cri-o://a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa" gracePeriod=30 Oct 30 18:59:20.420191 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.420160 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:59:20.607753 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.607719 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-metrics-certs\") pod \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " Oct 30 18:59:20.607959 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.607920 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc726b7-5486-4b21-aaa7-d5be63d3c467-kube-api-access-d6psv\" (UniqueName: \"kubernetes.io/projected/5bc726b7-5486-4b21-aaa7-d5be63d3c467-kube-api-access-d6psv\") pod \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " Oct 30 18:59:20.608046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.608030 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc726b7-5486-4b21-aaa7-d5be63d3c467-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc726b7-5486-4b21-aaa7-d5be63d3c467-openshift-service-ca\") pod \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " Oct 30 18:59:20.608148 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.608135 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls\") pod \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\" (UID: \"5bc726b7-5486-4b21-aaa7-d5be63d3c467\") " Oct 30 18:59:20.608431 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.608397 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc726b7-5486-4b21-aaa7-d5be63d3c467-openshift-service-ca" pod "5bc726b7-5486-4b21-aaa7-d5be63d3c467" (UID: "5bc726b7-5486-4b21-aaa7-d5be63d3c467"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:20.608847 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.608821 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc726b7-5486-4b21-aaa7-d5be63d3c467-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:20.610410 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.610387 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-metrics-certs" pod "5bc726b7-5486-4b21-aaa7-d5be63d3c467" (UID: "5bc726b7-5486-4b21-aaa7-d5be63d3c467"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:20.610978 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.610954 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls" pod "5bc726b7-5486-4b21-aaa7-d5be63d3c467" (UID: "5bc726b7-5486-4b21-aaa7-d5be63d3c467"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:20.611114 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.611092 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc726b7-5486-4b21-aaa7-d5be63d3c467-kube-api-access-d6psv" pod "5bc726b7-5486-4b21-aaa7-d5be63d3c467" (UID: "5bc726b7-5486-4b21-aaa7-d5be63d3c467"). InnerVolumeSpecName "kube-api-access-d6psv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:20.710682 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.710651 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:20.710682 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.710681 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bc726b7-5486-4b21-aaa7-d5be63d3c467-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:20.710897 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.710692 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6psv\" (UniqueName: \"kubernetes.io/projected/5bc726b7-5486-4b21-aaa7-d5be63d3c467-kube-api-access-d6psv\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:20.719739 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.719703 5959 generic.go:356] "Generic (PLEG): container finished" podID="5bc726b7-5486-4b21-aaa7-d5be63d3c467" containerID="a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa" exitCode=0 Oct 30 18:59:20.719898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.719764 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" Oct 30 18:59:20.719898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.719796 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" event={"ID":"5bc726b7-5486-4b21-aaa7-d5be63d3c467","Type":"ContainerDied","Data":"a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa"} Oct 30 18:59:20.719898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.719851 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8" event={"ID":"5bc726b7-5486-4b21-aaa7-d5be63d3c467","Type":"ContainerDied","Data":"ba18bd6d44a687f496c3d3d2f10d05afbc55243ffa7872b9f4d9bcf4320bd1d2"} Oct 30 18:59:20.719898 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.719895 5959 scope.go:122] "RemoveContainer" containerID="a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa" Oct 30 18:59:20.742358 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.742335 5959 scope.go:122] "RemoveContainer" containerID="a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa" Oct 30 18:59:20.742773 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:59:20.742743 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa\": container with ID starting with a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa not found: ID does not exist" containerID="a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa" Oct 30 18:59:20.742884 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.742779 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa"} err="failed to get container status \"a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa\": rpc error: code = NotFound desc = could not find container \"a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa\": container with ID starting with a8c79dc682bacdebd28a0c9f063b71b8ab78795383ebd83939e62481ffb87afa not found: ID does not exist" Oct 30 18:59:20.750083 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.750062 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8"] Oct 30 18:59:20.753304 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:20.753285 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54bc2b-7f98ccc875-vsln8"] Oct 30 18:59:21.798273 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:21.798235 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc726b7-5486-4b21-aaa7-d5be63d3c467" path="/var/lib/kubelet/pods/5bc726b7-5486-4b21-aaa7-d5be63d3c467/volumes" Oct 30 18:59:27.822397 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:59:27.822357 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:59:36.263749 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.263710 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf"] Oct 30 18:59:36.264347 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.263946 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" podUID="c823ebca-d0aa-4592-8d6e-a5273a34f3b3" containerName="maas-api" containerID="cri-o://0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854" gracePeriod=30 Oct 30 18:59:36.681629 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.677979 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:59:36.865670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.865563 5959 generic.go:356] "Generic (PLEG): container finished" podID="c823ebca-d0aa-4592-8d6e-a5273a34f3b3" containerID="0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854" exitCode=0 Oct 30 18:59:36.865670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.865625 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" Oct 30 18:59:36.865670 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.865648 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" event={"ID":"c823ebca-d0aa-4592-8d6e-a5273a34f3b3","Type":"ContainerDied","Data":"0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854"} Oct 30 18:59:36.865968 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.865715 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf" event={"ID":"c823ebca-d0aa-4592-8d6e-a5273a34f3b3","Type":"ContainerDied","Data":"1059b51725972db7da8f4e9d0c2fd4f5d4e77532c8abdf6382d02236597dcfd8"} Oct 30 18:59:36.865968 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.865737 5959 scope.go:122] "RemoveContainer" containerID="0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854" Oct 30 18:59:36.875421 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.875399 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-openshift-service-ca\") pod \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " Oct 30 18:59:36.875528 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.875482 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-kube-api-access-hvlg5\" (UniqueName: \"kubernetes.io/projected/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-kube-api-access-hvlg5\") pod \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " Oct 30 18:59:36.875573 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.875526 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-maas-api-tls\") pod \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " Oct 30 18:59:36.875642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.875616 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-metrics-certs\") pod \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\" (UID: \"c823ebca-d0aa-4592-8d6e-a5273a34f3b3\") " Oct 30 18:59:36.875817 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.875795 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-openshift-service-ca" pod "c823ebca-d0aa-4592-8d6e-a5273a34f3b3" (UID: "c823ebca-d0aa-4592-8d6e-a5273a34f3b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:36.876341 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.876275 5959 scope.go:122] "RemoveContainer" containerID="0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854" Oct 30 18:59:36.876432 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.876411 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:36.876664 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:59:36.876592 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854\": container with ID starting with 0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854 not found: ID does not exist" containerID="0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854" Oct 30 18:59:36.876800 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.876678 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854"} err="failed to get container status \"0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854\": rpc error: code = NotFound desc = could not find container \"0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854\": container with ID starting with 0b24bca29cc3514bc8d85b94a900336607f086615c609973d3c10ce96e233854 not found: ID does not exist" Oct 30 18:59:36.878150 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.878122 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-metrics-certs" pod "c823ebca-d0aa-4592-8d6e-a5273a34f3b3" (UID: "c823ebca-d0aa-4592-8d6e-a5273a34f3b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:36.878301 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.878276 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-maas-api-tls" pod "c823ebca-d0aa-4592-8d6e-a5273a34f3b3" (UID: "c823ebca-d0aa-4592-8d6e-a5273a34f3b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:36.878371 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.878316 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-kube-api-access-hvlg5" pod "c823ebca-d0aa-4592-8d6e-a5273a34f3b3" (UID: "c823ebca-d0aa-4592-8d6e-a5273a34f3b3"). InnerVolumeSpecName "kube-api-access-hvlg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:36.977925 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.977889 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvlg5\" (UniqueName: \"kubernetes.io/projected/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-kube-api-access-hvlg5\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:36.977925 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.977928 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:36.978130 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:36.977944 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c823ebca-d0aa-4592-8d6e-a5273a34f3b3-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:37.191742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:37.191716 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf"] Oct 30 18:59:37.196427 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:37.196386 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-655193-c8f4b9c5d-z92rf"] Oct 30 18:59:37.797712 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:37.797678 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c823ebca-d0aa-4592-8d6e-a5273a34f3b3" path="/var/lib/kubelet/pods/c823ebca-d0aa-4592-8d6e-a5273a34f3b3/volumes" Oct 30 18:59:45.066794 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:45.066720 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 18:59:45.425363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:45.422494 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5"] Oct 30 18:59:45.425363 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:45.423117 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" podUID="0b482d51-e94c-4e1e-8082-35e825cce973" containerName="maas-api" containerID="cri-o://c1b1f411be054401b76b5478dbda83d836d8b82972b87d0bd749ee4255e8e39f" gracePeriod=30 Oct 30 18:59:45.975632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:45.975131 5959 generic.go:356] "Generic (PLEG): container finished" podID="0b482d51-e94c-4e1e-8082-35e825cce973" containerID="c1b1f411be054401b76b5478dbda83d836d8b82972b87d0bd749ee4255e8e39f" exitCode=0 Oct 30 18:59:45.975632 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:45.975186 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" event={"ID":"0b482d51-e94c-4e1e-8082-35e825cce973","Type":"ContainerDied","Data":"c1b1f411be054401b76b5478dbda83d836d8b82972b87d0bd749ee4255e8e39f"} Oct 30 18:59:46.087184 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.087156 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:59:46.280983 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.280898 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-metrics-certs\") pod \"0b482d51-e94c-4e1e-8082-35e825cce973\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " Oct 30 18:59:46.281142 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.281047 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b482d51-e94c-4e1e-8082-35e825cce973-kube-api-access-cst8t\" (UniqueName: \"kubernetes.io/projected/0b482d51-e94c-4e1e-8082-35e825cce973-kube-api-access-cst8t\") pod \"0b482d51-e94c-4e1e-8082-35e825cce973\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " Oct 30 18:59:46.281142 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.281083 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b482d51-e94c-4e1e-8082-35e825cce973-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b482d51-e94c-4e1e-8082-35e825cce973-openshift-service-ca\") pod \"0b482d51-e94c-4e1e-8082-35e825cce973\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " Oct 30 18:59:46.281296 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.281278 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-maas-api-tls\") pod \"0b482d51-e94c-4e1e-8082-35e825cce973\" (UID: \"0b482d51-e94c-4e1e-8082-35e825cce973\") " Oct 30 18:59:46.281445 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.281416 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b482d51-e94c-4e1e-8082-35e825cce973-openshift-service-ca" pod "0b482d51-e94c-4e1e-8082-35e825cce973" (UID: "0b482d51-e94c-4e1e-8082-35e825cce973"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:46.281866 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.281830 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b482d51-e94c-4e1e-8082-35e825cce973-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:46.283155 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.283127 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-metrics-certs" pod "0b482d51-e94c-4e1e-8082-35e825cce973" (UID: "0b482d51-e94c-4e1e-8082-35e825cce973"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:46.283301 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.283284 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b482d51-e94c-4e1e-8082-35e825cce973-kube-api-access-cst8t" pod "0b482d51-e94c-4e1e-8082-35e825cce973" (UID: "0b482d51-e94c-4e1e-8082-35e825cce973"). InnerVolumeSpecName "kube-api-access-cst8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:46.283376 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.283355 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-maas-api-tls" pod "0b482d51-e94c-4e1e-8082-35e825cce973" (UID: "0b482d51-e94c-4e1e-8082-35e825cce973"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:46.383744 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.383707 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:46.383744 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.383743 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b482d51-e94c-4e1e-8082-35e825cce973-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:46.383996 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.383761 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cst8t\" (UniqueName: \"kubernetes.io/projected/0b482d51-e94c-4e1e-8082-35e825cce973-kube-api-access-cst8t\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:46.942148 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.941934 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z"] Oct 30 18:59:46.986742 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.986703 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" event={"ID":"0b482d51-e94c-4e1e-8082-35e825cce973","Type":"ContainerDied","Data":"4a398dbce7983caffdecc00e28c3901705f0e1d4d93926095eb806ca4cad9a4e"} Oct 30 18:59:46.986933 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.986778 5959 scope.go:122] "RemoveContainer" containerID="c1b1f411be054401b76b5478dbda83d836d8b82972b87d0bd749ee4255e8e39f" Oct 30 18:59:46.987379 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:46.986945 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5" Oct 30 18:59:47.016714 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:47.016681 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5"] Oct 30 18:59:47.023210 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:47.023188 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fe0612-555d9f4c8-r5fp5"] Oct 30 18:59:47.798673 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:47.798640 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b482d51-e94c-4e1e-8082-35e825cce973" path="/var/lib/kubelet/pods/0b482d51-e94c-4e1e-8082-35e825cce973/volumes" Oct 30 18:59:51.943226 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:51.943178 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" podUID="dd74c92b-bc79-431c-a713-7135937db8ff" containerName="payload-pre-processing" containerID="cri-o://289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741" gracePeriod=25 Oct 30 18:59:52.226186 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.226162 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:59:52.334642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.332280 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd74c92b-bc79-431c-a713-7135937db8ff-kube-api-access-cmz2h\" (UniqueName: \"kubernetes.io/projected/dd74c92b-bc79-431c-a713-7135937db8ff-kube-api-access-cmz2h\") pod \"dd74c92b-bc79-431c-a713-7135937db8ff\" (UID: \"dd74c92b-bc79-431c-a713-7135937db8ff\") " Oct 30 18:59:52.334642 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.332379 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd74c92b-bc79-431c-a713-7135937db8ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd74c92b-bc79-431c-a713-7135937db8ff-plugins-config-volume\") pod \"dd74c92b-bc79-431c-a713-7135937db8ff\" (UID: \"dd74c92b-bc79-431c-a713-7135937db8ff\") " Oct 30 18:59:52.351628 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.350891 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd74c92b-bc79-431c-a713-7135937db8ff-kube-api-access-cmz2h" pod "dd74c92b-bc79-431c-a713-7135937db8ff" (UID: "dd74c92b-bc79-431c-a713-7135937db8ff"). InnerVolumeSpecName "kube-api-access-cmz2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:52.374812 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.372863 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd74c92b-bc79-431c-a713-7135937db8ff-plugins-config-volume" pod "dd74c92b-bc79-431c-a713-7135937db8ff" (UID: "dd74c92b-bc79-431c-a713-7135937db8ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:52.434746 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.434717 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmz2h\" (UniqueName: \"kubernetes.io/projected/dd74c92b-bc79-431c-a713-7135937db8ff-kube-api-access-cmz2h\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:52.434746 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.434745 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd74c92b-bc79-431c-a713-7135937db8ff-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:52.894025 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.893989 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4"] Oct 30 18:59:52.894655 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:52.894581 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" containerID="cri-o://7c3a92b5a20cd532e447b5ca32384f46d92bfeccfb6d3e86aa59732fd49a4198" gracePeriod=30 Oct 30 18:59:53.027587 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.027548 5959 generic.go:356] "Generic (PLEG): container finished" podID="dd74c92b-bc79-431c-a713-7135937db8ff" containerID="289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741" exitCode=0 Oct 30 18:59:53.027587 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.027589 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" Oct 30 18:59:53.028141 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.027637 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" event={"ID":"dd74c92b-bc79-431c-a713-7135937db8ff","Type":"ContainerDied","Data":"289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741"} Oct 30 18:59:53.028141 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.027701 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" event={"ID":"dd74c92b-bc79-431c-a713-7135937db8ff","Type":"ContainerDied","Data":"3283fc97c5a1e20a7ee43337bc0b52dd644efb4144f2bfb9abc0cd07dff93066"} Oct 30 18:59:53.028141 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.027729 5959 scope.go:122] "RemoveContainer" containerID="289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741" Oct 30 18:59:53.127816 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.112010 5959 scope.go:122] "RemoveContainer" containerID="289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741" Oct 30 18:59:53.145754 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:59:53.128361 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741\": container with ID starting with 289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741 not found: ID does not exist" containerID="289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741" Oct 30 18:59:53.145754 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.128422 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741"} err="failed to get container status \"289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741\": rpc error: code = NotFound desc = could not find container \"289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741\": container with ID starting with 289365c5d8883084a524a447ed036a1fa307030a218b07734dffc1399306d741 not found: ID does not exist" Oct 30 18:59:53.153026 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.147081 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z"] Oct 30 18:59:53.153026 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.150433 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z"] Oct 30 18:59:53.220683 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.220565 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f778f5-f9577cdf5-84m4z" podUID="dd74c92b-bc79-431c-a713-7135937db8ff" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.92:50052\" within 1s: context deadline exceeded" Oct 30 18:59:53.220872 ip-10-0-133-96 kubenswrapper[5959]: W1030 18:59:53.220712 5959 logging.go:55] [core] [Channel #1226 SubChannel #1227]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.92:50052", ServerName: "10.134.0.92:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.92:50052: operation was canceled" Oct 30 18:59:53.797559 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:53.797517 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd74c92b-bc79-431c-a713-7135937db8ff" path="/var/lib/kubelet/pods/dd74c92b-bc79-431c-a713-7135937db8ff/volumes" Oct 30 18:59:57.781626 ip-10-0-133-96 kubenswrapper[5959]: E1030 18:59:57.781565 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 18:59:58.066225 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.066149 5959 generic.go:356] "Generic (PLEG): container finished" podID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerID="7c3a92b5a20cd532e447b5ca32384f46d92bfeccfb6d3e86aa59732fd49a4198" exitCode=0 Oct 30 18:59:58.066371 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.066302 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" event={"ID":"ccd42b08-75db-4201-8f3c-6c21fddf22eb","Type":"ContainerDied","Data":"7c3a92b5a20cd532e447b5ca32384f46d92bfeccfb6d3e86aa59732fd49a4198"} Oct 30 18:59:58.213359 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.213331 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:59:58.321578 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.321500 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-podinfo\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.321578 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.321568 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-kube-api-access-gmdrj\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-kube-api-access-gmdrj\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.321817 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.321682 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-envoy\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.321817 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.321772 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-token\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-token\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.321943 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.321812 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istiod-ca-cert\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.321943 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.321898 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-certs\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.322065 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.321953 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-socket\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.322065 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.321992 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-data\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.322158 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.322113 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-credential-socket\") pod \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\" (UID: \"ccd42b08-75db-4201-8f3c-6c21fddf22eb\") " Oct 30 18:59:58.322561 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.322317 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istiod-ca-cert" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:58.323046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.322752 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-certs" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:58.323046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.322786 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-socket" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:58.323046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.322990 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-data" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:58.323046 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.323018 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-credential-socket" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:58.323366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.323146 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:58.323366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.323167 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:58.323366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.323182 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:58.323366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.323222 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:58.323366 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.323237 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:58.324244 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.324220 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-kube-api-access-gmdrj" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "kube-api-access-gmdrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:58.324429 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.324399 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-token" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:58.324515 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.324449 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-envoy" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:58.324515 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.324489 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-podinfo" pod "ccd42b08-75db-4201-8f3c-6c21fddf22eb" (UID: "ccd42b08-75db-4201-8f3c-6c21fddf22eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:59:58.425430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.425312 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:58.425430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.425349 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmdrj\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-kube-api-access-gmdrj\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:58.425430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.425366 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:58.425430 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:58.425382 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd42b08-75db-4201-8f3c-6c21fddf22eb-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:59.074057 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:59.074026 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" Oct 30 18:59:59.074595 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:59.074057 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4" event={"ID":"ccd42b08-75db-4201-8f3c-6c21fddf22eb","Type":"ContainerDied","Data":"e19895f9bd11538b487b6f862d68fb5d098d0de5faa5aef14c1926e9d3b13ad2"} Oct 30 18:59:59.074595 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:59.074124 5959 scope.go:122] "RemoveContainer" containerID="7c3a92b5a20cd532e447b5ca32384f46d92bfeccfb6d3e86aa59732fd49a4198" Oct 30 18:59:59.098661 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:59.098633 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4"] Oct 30 18:59:59.101180 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:59.101157 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-v5fg4"] Oct 30 18:59:59.798910 ip-10-0-133-96 kubenswrapper[5959]: I1030 18:59:59.798878 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" path="/var/lib/kubelet/pods/ccd42b08-75db-4201-8f3c-6c21fddf22eb/volumes" Oct 30 19:00:00.145449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.145415 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c"] Oct 30 19:00:00.146240 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146219 5959 cpu_manager.go:414] "containerMap: removing container" podUID="0b482d51-e94c-4e1e-8082-35e825cce973" containerName="maas-api" Oct 30 19:00:00.146340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146241 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b482d51-e94c-4e1e-8082-35e825cce973" containerName="maas-api" Oct 30 19:00:00.146340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146255 5959 cpu_manager.go:414] "containerMap: removing container" podUID="c823ebca-d0aa-4592-8d6e-a5273a34f3b3" containerName="maas-api" Oct 30 19:00:00.146340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146264 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c823ebca-d0aa-4592-8d6e-a5273a34f3b3" containerName="maas-api" Oct 30 19:00:00.146340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146280 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5bc726b7-5486-4b21-aaa7-d5be63d3c467" containerName="maas-api" Oct 30 19:00:00.146340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146287 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc726b7-5486-4b21-aaa7-d5be63d3c467" containerName="maas-api" Oct 30 19:00:00.146340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146317 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" Oct 30 19:00:00.146340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146325 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" Oct 30 19:00:00.146340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146337 5959 cpu_manager.go:414] "containerMap: removing container" podUID="dd74c92b-bc79-431c-a713-7135937db8ff" containerName="payload-pre-processing" Oct 30 19:00:00.146759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146345 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd74c92b-bc79-431c-a713-7135937db8ff" containerName="payload-pre-processing" Oct 30 19:00:00.146759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146509 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b482d51-e94c-4e1e-8082-35e825cce973" containerName="maas-api" Oct 30 19:00:00.146759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146521 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c823ebca-d0aa-4592-8d6e-a5273a34f3b3" containerName="maas-api" Oct 30 19:00:00.146759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146532 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc726b7-5486-4b21-aaa7-d5be63d3c467" containerName="maas-api" Oct 30 19:00:00.146759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146545 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd42b08-75db-4201-8f3c-6c21fddf22eb" containerName="istio-proxy" Oct 30 19:00:00.146759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.146553 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd74c92b-bc79-431c-a713-7135937db8ff" containerName="payload-pre-processing" Oct 30 19:00:00.151840 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.151821 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:00.161931 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.161887 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\"" Oct 30 19:00:00.165076 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.165052 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c"] Oct 30 19:00:00.168273 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.168227 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl"] Oct 30 19:00:00.178894 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.178868 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:00.189028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.187489 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6"] Oct 30 19:00:00.194292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.194263 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl"] Oct 30 19:00:00.194436 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.194371 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:00.201620 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.201575 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6"] Oct 30 19:00:00.247343 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.247303 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rdj\" (UniqueName: \"kubernetes.io/projected/85147ab0-84fb-4d83-beeb-2a8e0325130c-kube-api-access-75rdj\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6\" (UID: \"85147ab0-84fb-4d83-beeb-2a8e0325130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:00.247519 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.247366 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2qw\" (UniqueName: \"kubernetes.io/projected/5cd79fee-1bed-47a0-aa08-7a8b559318a8-kube-api-access-9z2qw\") pod \"maas-api-key-cleanup-29889780-s6hnl\" (UID: \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:00.247519 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.247471 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/1d96b63f-fba8-4839-a8ce-7810d1676fc0-kube-api-access-vpvdh\") pod \"maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c\" (UID: \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:00.247673 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.247518 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d96b63f-fba8-4839-a8ce-7810d1676fc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c\" (UID: \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:00.247767 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.247746 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85147ab0-84fb-4d83-beeb-2a8e0325130c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6\" (UID: \"85147ab0-84fb-4d83-beeb-2a8e0325130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:00.247828 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.247802 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd79fee-1bed-47a0-aa08-7a8b559318a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-s6hnl\" (UID: \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:00.348946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.348914 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2qw\" (UniqueName: \"kubernetes.io/projected/5cd79fee-1bed-47a0-aa08-7a8b559318a8-kube-api-access-9z2qw\") pod \"maas-api-key-cleanup-29889780-s6hnl\" (UID: \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:00.349141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.348974 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/1d96b63f-fba8-4839-a8ce-7810d1676fc0-kube-api-access-vpvdh\") pod \"maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c\" (UID: \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:00.349141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.349016 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d96b63f-fba8-4839-a8ce-7810d1676fc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c\" (UID: \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:00.349141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.349086 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85147ab0-84fb-4d83-beeb-2a8e0325130c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6\" (UID: \"85147ab0-84fb-4d83-beeb-2a8e0325130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:00.349141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.349115 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd79fee-1bed-47a0-aa08-7a8b559318a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-s6hnl\" (UID: \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:00.349333 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.349166 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rdj\" (UniqueName: \"kubernetes.io/projected/85147ab0-84fb-4d83-beeb-2a8e0325130c-kube-api-access-75rdj\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6\" (UID: \"85147ab0-84fb-4d83-beeb-2a8e0325130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:00.349814 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.349795 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85147ab0-84fb-4d83-beeb-2a8e0325130c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6\" (UID: \"85147ab0-84fb-4d83-beeb-2a8e0325130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:00.349866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.349815 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd79fee-1bed-47a0-aa08-7a8b559318a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-s6hnl\" (UID: \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:00.349866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.349835 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d96b63f-fba8-4839-a8ce-7810d1676fc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c\" (UID: \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:00.357642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.357584 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/1d96b63f-fba8-4839-a8ce-7810d1676fc0-kube-api-access-vpvdh\") pod \"maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c\" (UID: \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:00.357746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.357664 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rdj\" (UniqueName: \"kubernetes.io/projected/85147ab0-84fb-4d83-beeb-2a8e0325130c-kube-api-access-75rdj\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6\" (UID: \"85147ab0-84fb-4d83-beeb-2a8e0325130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:00.357746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.357719 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2qw\" (UniqueName: \"kubernetes.io/projected/5cd79fee-1bed-47a0-aa08-7a8b559318a8-kube-api-access-9z2qw\") pod \"maas-api-key-cleanup-29889780-s6hnl\" (UID: \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:00.469467 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.469390 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:00.511063 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.511037 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:00.521989 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.519933 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:00.692685 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.692643 5959 scope.go:122] "RemoveContainer" containerID="051cee7a3811f41829b9c869904b721a6a9420af9a839913b391626050633303" Oct 30 19:00:00.849763 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:00.847664 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c"] Oct 30 19:00:01.067389 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:00:01.067349 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd79fee_1bed_47a0_aa08_7a8b559318a8.slice/crio-c2a3890141be2b18e1e4739b9e5a6166f385c6c89949b91a6803cfc1bbfbcc0e WatchSource:0}: Error finding container c2a3890141be2b18e1e4739b9e5a6166f385c6c89949b91a6803cfc1bbfbcc0e: Status 404 returned error can't find the container with id c2a3890141be2b18e1e4739b9e5a6166f385c6c89949b91a6803cfc1bbfbcc0e Oct 30 19:00:01.068272 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:01.068245 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl"] Oct 30 19:00:01.089018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:01.088988 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6"] Oct 30 19:00:01.090849 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:01.090805 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" event={"ID":"1d96b63f-fba8-4839-a8ce-7810d1676fc0","Type":"ContainerStarted","Data":"4365fcaeea1cba767cc540126b5e9482dcf40f1a1ec5d401dd7f5206891cb667"} Oct 30 19:00:01.091443 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:00:01.091376 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85147ab0_84fb_4d83_beeb_2a8e0325130c.slice/crio-f1c95e832b8062a62e4426d4becf85b8ed12bcf67d23269b75ce4f43986e2a26 WatchSource:0}: Error finding container f1c95e832b8062a62e4426d4becf85b8ed12bcf67d23269b75ce4f43986e2a26: Status 404 returned error can't find the container with id f1c95e832b8062a62e4426d4becf85b8ed12bcf67d23269b75ce4f43986e2a26 Oct 30 19:00:01.092843 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:01.092813 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" event={"ID":"5cd79fee-1bed-47a0-aa08-7a8b559318a8","Type":"ContainerStarted","Data":"c2a3890141be2b18e1e4739b9e5a6166f385c6c89949b91a6803cfc1bbfbcc0e"} Oct 30 19:00:02.103584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:02.103546 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" event={"ID":"85147ab0-84fb-4d83-beeb-2a8e0325130c","Type":"ContainerStarted","Data":"866ed8c161029ee73fd17f38efa6bdd3592fb9b49b49a547d10876ea873e56bd"} Oct 30 19:00:02.104007 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:02.103666 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" event={"ID":"85147ab0-84fb-4d83-beeb-2a8e0325130c","Type":"ContainerStarted","Data":"f1c95e832b8062a62e4426d4becf85b8ed12bcf67d23269b75ce4f43986e2a26"} Oct 30 19:00:02.105741 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:02.105713 5959 generic.go:356] "Generic (PLEG): container finished" podID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerID="165156a3725934cb5a6e080f83a45435ed06fc0e2c5162ae5f93b9bc7190dce3" exitCode=60 Oct 30 19:00:02.105869 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:02.105789 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" event={"ID":"1d96b63f-fba8-4839-a8ce-7810d1676fc0","Type":"ContainerDied","Data":"165156a3725934cb5a6e080f83a45435ed06fc0e2c5162ae5f93b9bc7190dce3"} Oct 30 19:00:02.106033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:02.106015 5959 scope.go:122] "RemoveContainer" containerID="165156a3725934cb5a6e080f83a45435ed06fc0e2c5162ae5f93b9bc7190dce3" Oct 30 19:00:02.107731 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:02.107598 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" event={"ID":"5cd79fee-1bed-47a0-aa08-7a8b559318a8","Type":"ContainerStarted","Data":"26913a12d199ad9f5d391930f3930d492d140665f0ef4c356cd9a178cca7ec72"} Oct 30 19:00:02.107907 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:02.107890 5959 scope.go:122] "RemoveContainer" containerID="26913a12d199ad9f5d391930f3930d492d140665f0ef4c356cd9a178cca7ec72" Oct 30 19:00:02.122312 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:02.122237 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" podStartSLOduration=2.122220764 podStartE2EDuration="2.122220764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.118940034 +0000 UTC m=+1382.447351420" watchObservedRunningTime="2026-10-30 19:00:02.122220764 +0000 UTC m=+1382.450632149" Oct 30 19:00:03.118826 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.118793 5959 generic.go:356] "Generic (PLEG): container finished" podID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerID="866ed8c161029ee73fd17f38efa6bdd3592fb9b49b49a547d10876ea873e56bd" exitCode=60 Oct 30 19:00:03.119192 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.118876 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" event={"ID":"85147ab0-84fb-4d83-beeb-2a8e0325130c","Type":"ContainerDied","Data":"866ed8c161029ee73fd17f38efa6bdd3592fb9b49b49a547d10876ea873e56bd"} Oct 30 19:00:03.119259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.119192 5959 scope.go:122] "RemoveContainer" containerID="866ed8c161029ee73fd17f38efa6bdd3592fb9b49b49a547d10876ea873e56bd" Oct 30 19:00:03.121339 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.121314 5959 generic.go:356] "Generic (PLEG): container finished" podID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerID="1cd0ccf12b5cfcc4965a7bdf7f6dbeb69de40cdca864acf7eab05151ff405697" exitCode=60 Oct 30 19:00:03.121460 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.121395 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" event={"ID":"1d96b63f-fba8-4839-a8ce-7810d1676fc0","Type":"ContainerDied","Data":"1cd0ccf12b5cfcc4965a7bdf7f6dbeb69de40cdca864acf7eab05151ff405697"} Oct 30 19:00:03.121529 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.121458 5959 scope.go:122] "RemoveContainer" containerID="165156a3725934cb5a6e080f83a45435ed06fc0e2c5162ae5f93b9bc7190dce3" Oct 30 19:00:03.121703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.121680 5959 scope.go:122] "RemoveContainer" containerID="1cd0ccf12b5cfcc4965a7bdf7f6dbeb69de40cdca864acf7eab05151ff405697" Oct 30 19:00:03.122498 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:03.122419 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c_odh-ai-gateway-infra(1d96b63f-fba8-4839-a8ce-7810d1676fc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" Oct 30 19:00:03.123535 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.123512 5959 generic.go:356] "Generic (PLEG): container finished" podID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerID="26913a12d199ad9f5d391930f3930d492d140665f0ef4c356cd9a178cca7ec72" exitCode=60 Oct 30 19:00:03.123631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.123588 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" event={"ID":"5cd79fee-1bed-47a0-aa08-7a8b559318a8","Type":"ContainerDied","Data":"26913a12d199ad9f5d391930f3930d492d140665f0ef4c356cd9a178cca7ec72"} Oct 30 19:00:03.123682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.123658 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" event={"ID":"5cd79fee-1bed-47a0-aa08-7a8b559318a8","Type":"ContainerStarted","Data":"baf42a3c54e3006be6caf993d610f2b943dbb7b8d37b69e1337ac42cc8562093"} Oct 30 19:00:03.123858 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.123842 5959 scope.go:122] "RemoveContainer" containerID="baf42a3c54e3006be6caf993d610f2b943dbb7b8d37b69e1337ac42cc8562093" Oct 30 19:00:03.124094 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:03.124077 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-s6hnl_odh-ai-gateway-infra(5cd79fee-1bed-47a0-aa08-7a8b559318a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" Oct 30 19:00:03.243731 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:03.243712 5959 scope.go:122] "RemoveContainer" containerID="26913a12d199ad9f5d391930f3930d492d140665f0ef4c356cd9a178cca7ec72" Oct 30 19:00:04.131938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:04.131913 5959 scope.go:122] "RemoveContainer" containerID="1cd0ccf12b5cfcc4965a7bdf7f6dbeb69de40cdca864acf7eab05151ff405697" Oct 30 19:00:04.132300 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:04.132190 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c_odh-ai-gateway-infra(1d96b63f-fba8-4839-a8ce-7810d1676fc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" Oct 30 19:00:04.133118 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:04.133098 5959 generic.go:356] "Generic (PLEG): container finished" podID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerID="baf42a3c54e3006be6caf993d610f2b943dbb7b8d37b69e1337ac42cc8562093" exitCode=60 Oct 30 19:00:04.133259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:04.133236 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" event={"ID":"5cd79fee-1bed-47a0-aa08-7a8b559318a8","Type":"ContainerDied","Data":"baf42a3c54e3006be6caf993d610f2b943dbb7b8d37b69e1337ac42cc8562093"} Oct 30 19:00:04.133426 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:04.133411 5959 scope.go:122] "RemoveContainer" containerID="baf42a3c54e3006be6caf993d610f2b943dbb7b8d37b69e1337ac42cc8562093" Oct 30 19:00:04.133713 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:04.133693 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-s6hnl_odh-ai-gateway-infra(5cd79fee-1bed-47a0-aa08-7a8b559318a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" Oct 30 19:00:04.135367 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:04.135330 5959 generic.go:356] "Generic (PLEG): container finished" podID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerID="effe4f02ea685b4f094dc14575fb63840425a523526e7d59abd83be0b4db6802" exitCode=60 Oct 30 19:00:04.135443 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:04.135417 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" event={"ID":"85147ab0-84fb-4d83-beeb-2a8e0325130c","Type":"ContainerDied","Data":"effe4f02ea685b4f094dc14575fb63840425a523526e7d59abd83be0b4db6802"} Oct 30 19:00:04.135495 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:04.135464 5959 scope.go:122] "RemoveContainer" containerID="866ed8c161029ee73fd17f38efa6bdd3592fb9b49b49a547d10876ea873e56bd" Oct 30 19:00:04.135617 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:04.135587 5959 scope.go:122] "RemoveContainer" containerID="effe4f02ea685b4f094dc14575fb63840425a523526e7d59abd83be0b4db6802" Oct 30 19:00:04.135880 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:04.135864 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6_odh-ai-gateway-infra(85147ab0-84fb-4d83-beeb-2a8e0325130c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" Oct 30 19:00:05.160853 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:05.160818 5959 scope.go:122] "RemoveContainer" containerID="effe4f02ea685b4f094dc14575fb63840425a523526e7d59abd83be0b4db6802" Oct 30 19:00:05.161340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:05.161157 5959 scope.go:122] "RemoveContainer" containerID="baf42a3c54e3006be6caf993d610f2b943dbb7b8d37b69e1337ac42cc8562093" Oct 30 19:00:05.161340 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:05.161174 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6_odh-ai-gateway-infra(85147ab0-84fb-4d83-beeb-2a8e0325130c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" Oct 30 19:00:05.161488 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:05.161448 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-s6hnl_odh-ai-gateway-infra(5cd79fee-1bed-47a0-aa08-7a8b559318a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" Oct 30 19:00:16.789939 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:16.789905 5959 scope.go:122] "RemoveContainer" containerID="1cd0ccf12b5cfcc4965a7bdf7f6dbeb69de40cdca864acf7eab05151ff405697" Oct 30 19:00:16.790452 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:16.790059 5959 scope.go:122] "RemoveContainer" containerID="baf42a3c54e3006be6caf993d610f2b943dbb7b8d37b69e1337ac42cc8562093" Oct 30 19:00:17.249100 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:17.249067 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" event={"ID":"1d96b63f-fba8-4839-a8ce-7810d1676fc0","Type":"ContainerStarted","Data":"c2ec45ecb43dd4ca41f56a6308f1c2f71680972863cd12198eb4b901b8def489"} Oct 30 19:00:17.251035 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:17.251000 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" event={"ID":"5cd79fee-1bed-47a0-aa08-7a8b559318a8","Type":"ContainerStarted","Data":"bdf80c5bfb99160f8bd3cdb68a1ac41a5986c566738e95ba1a1b53e33e278128"} Oct 30 19:00:17.273114 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:17.273068 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" podStartSLOduration=17.273055316 podStartE2EDuration="17.273055316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:17.270899925 +0000 UTC m=+1397.599311314" watchObservedRunningTime="2026-10-30 19:00:17.273055316 +0000 UTC m=+1397.601466691" Oct 30 19:00:17.296686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:17.296638 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" podStartSLOduration=17.296620875 podStartE2EDuration="17.296620875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:17.294496263 +0000 UTC m=+1397.622907642" watchObservedRunningTime="2026-10-30 19:00:17.296620875 +0000 UTC m=+1397.625032261" Oct 30 19:00:17.789654 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:17.789596 5959 scope.go:122] "RemoveContainer" containerID="effe4f02ea685b4f094dc14575fb63840425a523526e7d59abd83be0b4db6802" Oct 30 19:00:18.260009 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.259978 5959 generic.go:356] "Generic (PLEG): container finished" podID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerID="c2ec45ecb43dd4ca41f56a6308f1c2f71680972863cd12198eb4b901b8def489" exitCode=60 Oct 30 19:00:18.260424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.260047 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" event={"ID":"1d96b63f-fba8-4839-a8ce-7810d1676fc0","Type":"ContainerDied","Data":"c2ec45ecb43dd4ca41f56a6308f1c2f71680972863cd12198eb4b901b8def489"} Oct 30 19:00:18.260424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.260108 5959 scope.go:122] "RemoveContainer" containerID="1cd0ccf12b5cfcc4965a7bdf7f6dbeb69de40cdca864acf7eab05151ff405697" Oct 30 19:00:18.260565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.260520 5959 scope.go:122] "RemoveContainer" containerID="c2ec45ecb43dd4ca41f56a6308f1c2f71680972863cd12198eb4b901b8def489" Oct 30 19:00:18.260925 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:18.260903 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c_odh-ai-gateway-infra(1d96b63f-fba8-4839-a8ce-7810d1676fc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" Oct 30 19:00:18.261989 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.261968 5959 generic.go:356] "Generic (PLEG): container finished" podID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerID="bdf80c5bfb99160f8bd3cdb68a1ac41a5986c566738e95ba1a1b53e33e278128" exitCode=60 Oct 30 19:00:18.262076 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.262040 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" event={"ID":"5cd79fee-1bed-47a0-aa08-7a8b559318a8","Type":"ContainerDied","Data":"bdf80c5bfb99160f8bd3cdb68a1ac41a5986c566738e95ba1a1b53e33e278128"} Oct 30 19:00:18.262419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.262399 5959 scope.go:122] "RemoveContainer" containerID="bdf80c5bfb99160f8bd3cdb68a1ac41a5986c566738e95ba1a1b53e33e278128" Oct 30 19:00:18.262763 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:18.262745 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-s6hnl_odh-ai-gateway-infra(5cd79fee-1bed-47a0-aa08-7a8b559318a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" Oct 30 19:00:18.264051 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.263997 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" event={"ID":"85147ab0-84fb-4d83-beeb-2a8e0325130c","Type":"ContainerStarted","Data":"9fd565d3ea54e91c69ff07270cbc272f4d8cf1c8ae4b66ca46a2bc3d2b7e1652"} Oct 30 19:00:18.273173 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.273157 5959 scope.go:122] "RemoveContainer" containerID="baf42a3c54e3006be6caf993d610f2b943dbb7b8d37b69e1337ac42cc8562093" Oct 30 19:00:18.291500 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.291479 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c"] Oct 30 19:00:18.307538 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:18.307504 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl"] Oct 30 19:00:19.276531 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.276487 5959 generic.go:356] "Generic (PLEG): container finished" podID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerID="9fd565d3ea54e91c69ff07270cbc272f4d8cf1c8ae4b66ca46a2bc3d2b7e1652" exitCode=60 Oct 30 19:00:19.276889 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.276556 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" event={"ID":"85147ab0-84fb-4d83-beeb-2a8e0325130c","Type":"ContainerDied","Data":"9fd565d3ea54e91c69ff07270cbc272f4d8cf1c8ae4b66ca46a2bc3d2b7e1652"} Oct 30 19:00:19.276889 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.276695 5959 scope.go:122] "RemoveContainer" containerID="effe4f02ea685b4f094dc14575fb63840425a523526e7d59abd83be0b4db6802" Oct 30 19:00:19.277204 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.277182 5959 scope.go:122] "RemoveContainer" containerID="9fd565d3ea54e91c69ff07270cbc272f4d8cf1c8ae4b66ca46a2bc3d2b7e1652" Oct 30 19:00:19.277635 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:19.277551 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6_odh-ai-gateway-infra(85147ab0-84fb-4d83-beeb-2a8e0325130c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" Oct 30 19:00:19.307030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.307000 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6"] Oct 30 19:00:19.466568 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.465836 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:19.503648 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.503584 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:19.605516 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.605446 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d96b63f-fba8-4839-a8ce-7810d1676fc0-kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/1d96b63f-fba8-4839-a8ce-7810d1676fc0-kube-api-access-vpvdh\") pod \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\" (UID: \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\") " Oct 30 19:00:19.605516 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.605490 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd79fee-1bed-47a0-aa08-7a8b559318a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd79fee-1bed-47a0-aa08-7a8b559318a8-maas-api-service-ca\") pod \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\" (UID: \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\") " Oct 30 19:00:19.605783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.605538 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd79fee-1bed-47a0-aa08-7a8b559318a8-kube-api-access-9z2qw\" (UniqueName: \"kubernetes.io/projected/5cd79fee-1bed-47a0-aa08-7a8b559318a8-kube-api-access-9z2qw\") pod \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\" (UID: \"5cd79fee-1bed-47a0-aa08-7a8b559318a8\") " Oct 30 19:00:19.605783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.605661 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d96b63f-fba8-4839-a8ce-7810d1676fc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d96b63f-fba8-4839-a8ce-7810d1676fc0-maas-api-service-ca\") pod \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\" (UID: \"1d96b63f-fba8-4839-a8ce-7810d1676fc0\") " Oct 30 19:00:19.605968 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.605941 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd79fee-1bed-47a0-aa08-7a8b559318a8-maas-api-service-ca" pod "5cd79fee-1bed-47a0-aa08-7a8b559318a8" (UID: "5cd79fee-1bed-47a0-aa08-7a8b559318a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.606092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.606010 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d96b63f-fba8-4839-a8ce-7810d1676fc0-maas-api-service-ca" pod "1d96b63f-fba8-4839-a8ce-7810d1676fc0" (UID: "1d96b63f-fba8-4839-a8ce-7810d1676fc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.606206 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.606188 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd79fee-1bed-47a0-aa08-7a8b559318a8-maas-api-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.606258 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.606213 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d96b63f-fba8-4839-a8ce-7810d1676fc0-maas-api-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.607825 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.607807 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd79fee-1bed-47a0-aa08-7a8b559318a8-kube-api-access-9z2qw" pod "5cd79fee-1bed-47a0-aa08-7a8b559318a8" (UID: "5cd79fee-1bed-47a0-aa08-7a8b559318a8"). InnerVolumeSpecName "kube-api-access-9z2qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.607897 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.607843 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d96b63f-fba8-4839-a8ce-7810d1676fc0-kube-api-access-vpvdh" pod "1d96b63f-fba8-4839-a8ce-7810d1676fc0" (UID: "1d96b63f-fba8-4839-a8ce-7810d1676fc0"). InnerVolumeSpecName "kube-api-access-vpvdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.707946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.707913 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/1d96b63f-fba8-4839-a8ce-7810d1676fc0-kube-api-access-vpvdh\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.707946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:19.707943 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z2qw\" (UniqueName: \"kubernetes.io/projected/5cd79fee-1bed-47a0-aa08-7a8b559318a8-kube-api-access-9z2qw\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:20.284388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.284359 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" Oct 30 19:00:20.284915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.284397 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl" event={"ID":"5cd79fee-1bed-47a0-aa08-7a8b559318a8","Type":"ContainerDied","Data":"c2a3890141be2b18e1e4739b9e5a6166f385c6c89949b91a6803cfc1bbfbcc0e"} Oct 30 19:00:20.284915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.284461 5959 scope.go:122] "RemoveContainer" containerID="bdf80c5bfb99160f8bd3cdb68a1ac41a5986c566738e95ba1a1b53e33e278128" Oct 30 19:00:20.288277 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.288254 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" event={"ID":"1d96b63f-fba8-4839-a8ce-7810d1676fc0","Type":"ContainerDied","Data":"4365fcaeea1cba767cc540126b5e9482dcf40f1a1ec5d401dd7f5206891cb667"} Oct 30 19:00:20.288369 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.288314 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c" Oct 30 19:00:20.295885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.295867 5959 scope.go:122] "RemoveContainer" containerID="c2ec45ecb43dd4ca41f56a6308f1c2f71680972863cd12198eb4b901b8def489" Oct 30 19:00:20.303927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.303898 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl"] Oct 30 19:00:20.309815 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.309793 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-s6hnl"] Oct 30 19:00:20.344404 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.344365 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c"] Oct 30 19:00:20.353716 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.353686 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2889dd-29889780-b4k5c"] Oct 30 19:00:20.452745 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.452719 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:20.518122 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.518095 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85147ab0-84fb-4d83-beeb-2a8e0325130c-kube-api-access-75rdj\" (UniqueName: \"kubernetes.io/projected/85147ab0-84fb-4d83-beeb-2a8e0325130c-kube-api-access-75rdj\") pod \"85147ab0-84fb-4d83-beeb-2a8e0325130c\" (UID: \"85147ab0-84fb-4d83-beeb-2a8e0325130c\") " Oct 30 19:00:20.518302 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.518256 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85147ab0-84fb-4d83-beeb-2a8e0325130c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85147ab0-84fb-4d83-beeb-2a8e0325130c-maas-api-service-ca\") pod \"85147ab0-84fb-4d83-beeb-2a8e0325130c\" (UID: \"85147ab0-84fb-4d83-beeb-2a8e0325130c\") " Oct 30 19:00:20.518585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.518565 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85147ab0-84fb-4d83-beeb-2a8e0325130c-maas-api-service-ca" pod "85147ab0-84fb-4d83-beeb-2a8e0325130c" (UID: "85147ab0-84fb-4d83-beeb-2a8e0325130c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:20.518742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.518728 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85147ab0-84fb-4d83-beeb-2a8e0325130c-maas-api-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:20.520425 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.520396 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85147ab0-84fb-4d83-beeb-2a8e0325130c-kube-api-access-75rdj" pod "85147ab0-84fb-4d83-beeb-2a8e0325130c" (UID: "85147ab0-84fb-4d83-beeb-2a8e0325130c"). InnerVolumeSpecName "kube-api-access-75rdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:20.620090 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:20.620007 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rdj\" (UniqueName: \"kubernetes.io/projected/85147ab0-84fb-4d83-beeb-2a8e0325130c-kube-api-access-75rdj\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:21.299369 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:21.299330 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" event={"ID":"85147ab0-84fb-4d83-beeb-2a8e0325130c","Type":"ContainerDied","Data":"f1c95e832b8062a62e4426d4becf85b8ed12bcf67d23269b75ce4f43986e2a26"} Oct 30 19:00:21.299369 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:21.299367 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6" Oct 30 19:00:21.299891 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:21.299438 5959 scope.go:122] "RemoveContainer" containerID="9fd565d3ea54e91c69ff07270cbc272f4d8cf1c8ae4b66ca46a2bc3d2b7e1652" Oct 30 19:00:21.322466 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:21.322439 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6"] Oct 30 19:00:21.331875 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:21.331521 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889780-qt8d6"] Oct 30 19:00:21.802260 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:21.802231 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" path="/var/lib/kubelet/pods/1d96b63f-fba8-4839-a8ce-7810d1676fc0/volumes" Oct 30 19:00:21.802805 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:21.802790 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" path="/var/lib/kubelet/pods/5cd79fee-1bed-47a0-aa08-7a8b559318a8/volumes" Oct 30 19:00:21.803216 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:21.803200 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" path="/var/lib/kubelet/pods/85147ab0-84fb-4d83-beeb-2a8e0325130c/volumes" Oct 30 19:00:27.781647 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:27.781594 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:00:35.773381 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:35.773346 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:00:38.545637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:38.544753 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:38.590098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:38.588817 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:38.817874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:38.816689 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:39.030750 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.030698 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:00:39.031392 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.031360 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:00:39.083850 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.083739 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:39.521630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.507014 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 19:00:39.559484 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.559442 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:39.560573 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.560551 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:00:39.602078 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.602027 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:39.621634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.611527 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:00:39.901709 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:39.883721 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:40.115827 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.115788 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:40.133278 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.123118 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:00:40.146209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.130375 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:00:40.153679 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.151704 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 19:00:40.153679 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.152041 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 19:00:40.153679 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.152097 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:00:40.341631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.340011 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:00:40.341631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.340096 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:00:40.341631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.340135 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:00:40.341631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.340170 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:00:40.417166 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.417076 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:00:40.420452 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.420434 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:00:40.420680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.420668 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:00:40.425637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.422583 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:00:40.432632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.431955 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:00:40.455599 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:40.455572 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:00:44.315997 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.315956 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s"] Oct 30 19:00:44.316768 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316746 5959 cpu_manager.go:414] "containerMap: removing container" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:00:44.316885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316769 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:00:44.316885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316786 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:00:44.316885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316793 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:00:44.316885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316838 5959 cpu_manager.go:414] "containerMap: removing container" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.316885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316845 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.316885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316860 5959 cpu_manager.go:414] "containerMap: removing container" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.316885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316867 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316896 5959 cpu_manager.go:414] "containerMap: removing container" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316903 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316919 5959 cpu_manager.go:414] "containerMap: removing container" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316926 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316935 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.316941 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317130 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317142 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317152 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317165 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d96b63f-fba8-4839-a8ce-7810d1676fc0" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317174 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317184 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317195 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317209 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:00:44.317234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.317219 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:00:44.322563 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.322537 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.368158 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.368129 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s"] Oct 30 19:00:44.394770 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.394731 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.395038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.395010 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.395170 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.395103 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6l88\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-kube-api-access-m6l88\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.395241 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.395201 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.395310 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.395252 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.395310 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.395284 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.395412 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.395331 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.395412 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.395377 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.395518 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.395408 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.496972 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.496942 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497167 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497002 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6l88\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-kube-api-access-m6l88\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497167 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497047 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497167 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497072 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497167 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497097 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497167 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497139 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497180 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497204 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497346 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497649 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497469 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497649 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497510 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497769 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497715 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.497769 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.497721 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.498135 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.498112 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.499648 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.499590 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.499775 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.499763 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.508956 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.508936 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.509224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.509202 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6l88\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-kube-api-access-m6l88\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.639825 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.639781 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:44.945290 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.945262 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s"] Oct 30 19:00:44.946876 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:00:44.946845 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58ad637_7f3a_49c6_93e4_97a9ea5c584b.slice/crio-0dc46c1d9abd7f4b209d6b02b6198d6a5bf853b22b4dc39b6920954e3cae62c5 WatchSource:0}: Error finding container 0dc46c1d9abd7f4b209d6b02b6198d6a5bf853b22b4dc39b6920954e3cae62c5: Status 404 returned error can't find the container with id 0dc46c1d9abd7f4b209d6b02b6198d6a5bf853b22b4dc39b6920954e3cae62c5 Oct 30 19:00:44.949543 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.949508 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:00:44.949651 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.949576 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:00:44.949698 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:44.949663 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:00:45.193060 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:45.193024 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" event={"ID":"a58ad637-7f3a-49c6-93e4-97a9ea5c584b","Type":"ContainerStarted","Data":"f566bc1d57c95bfac8ad96a95230550b27112ba690470168397aab5f4c5cd258"} Oct 30 19:00:45.193256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:45.193077 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" event={"ID":"a58ad637-7f3a-49c6-93e4-97a9ea5c584b","Type":"ContainerStarted","Data":"0dc46c1d9abd7f4b209d6b02b6198d6a5bf853b22b4dc39b6920954e3cae62c5"} Oct 30 19:00:45.223937 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:45.223843 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" podStartSLOduration=1.223823755 podStartE2EDuration="1.223823755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:45.216664027 +0000 UTC m=+1425.545075413" watchObservedRunningTime="2026-10-30 19:00:45.223823755 +0000 UTC m=+1425.552235143" Oct 30 19:00:45.639936 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:45.639903 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:46.641833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:46.641797 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:00:46.642229 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:46.641859 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:00:47.640127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:47.640093 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:00:47.640353 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:47.640157 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:00:48.641582 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:48.641545 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:00:48.642111 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:48.641620 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:00:49.640828 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:49.640789 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:00:49.641010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:49.640847 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:00:50.640950 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:50.640914 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:00:50.641379 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:50.640969 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:00:51.640912 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:51.640876 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:00:51.641082 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:51.640941 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:00:52.644725 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:52.644695 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:52.645095 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:52.644947 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:52.645724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:00:52.645706 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:00:57.815759 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:00:57.815720 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:01:01.835882 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:01.835838 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:01.898340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:01.898294 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:02.226643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.225726 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:01:02.226643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.225798 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:02.395761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.395704 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:02.433215 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.433167 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:02.436019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.435981 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:01:02.436490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.436463 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:01:02.436897 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.436874 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 19:01:02.440627 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.437243 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 19:01:02.479290 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.479176 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:01:02.518093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.518047 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:01:02.518398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.518365 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:02.837684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.836708 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:02.837684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.837129 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:01:02.906729 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.906669 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:01:02.930649 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:02.907827 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:01:03.493668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.442806 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:03.493668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.443420 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:01:03.493668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.444530 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:03.528878 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.528382 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:01:03.530124 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.530102 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:01:03.530369 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.530341 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:01:03.620147 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.591196 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:01:03.620756 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.591231 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:01:03.654657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.625010 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:01:03.654657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.625072 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:01:03.654657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.591266 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 19:01:03.654657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.592408 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:03.654657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.626162 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:01:03.654657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.626208 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:01:03.823643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.823471 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:01:03.823909 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.823892 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:01:03.833624 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.827121 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:01:03.833624 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:03.827939 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:01:04.081205 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:04.081121 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:01:04.082049 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:04.082030 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:01:04.082225 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:04.082215 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:01:04.084225 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:04.083813 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:01:04.084225 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:04.083891 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:01:15.787258 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:15.787165 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:01:18.838716 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:18.838653 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:18.842954 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:18.842908 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:01:18.868030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:18.867982 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:19.096690 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:19.045138 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:19.107468 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:19.107422 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 19:01:19.137696 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:19.126988 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:19.189630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:19.188474 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:27.781848 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:01:27.781804 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:01:29.857631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:29.834826 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:29.857631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:29.835412 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:29.944645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:29.944571 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:30.022289 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.022246 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:30.241675 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.062043 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:01:30.268942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.089328 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:01:30.308687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.091709 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:30.308687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.091751 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 19:01:30.308687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.121394 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:30.395636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.382759 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:30.996791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.996738 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:30.998791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.998751 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:30.999472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:30.999450 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:01:31.044471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.043912 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:31.045508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.045482 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:01:31.243977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.243928 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:31.246214 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.246185 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:01:31.341882 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.276265 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:31.356631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.300903 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:31.384652 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.302015 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:31.384652 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.302048 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:01:31.397640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.396621 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:01:31.397640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.396690 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:01:31.397640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.396745 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:01:31.397640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.396784 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:01:31.426803 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.426747 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:31.428902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:31.428875 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:01:32.001577 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:32.001546 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:01:32.034181 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:32.034146 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:01:32.034472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:32.034457 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:01:32.039396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:32.039368 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:01:32.039684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:32.039669 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:01:32.043660 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:32.043596 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:01:32.175299 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:32.175272 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:01:32.187629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:32.187585 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:01:43.129620 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:43.129577 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:01:45.890631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:45.890104 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:01:45.916632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:45.914021 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:46.039836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:46.039768 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:46.209690 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:46.209193 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:46.509690 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:46.503711 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:46.955659 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:46.913362 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:46.955659 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:46.929758 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:01:46.955659 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:46.930379 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:01:47.214824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.214708 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 19:01:47.238634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.215819 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:01:47.239651 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.227727 5959 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:01:47.254752 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.254708 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:47.609870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.609770 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:47.933067 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.933017 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:47.933841 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.933812 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:01:47.936142 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.935370 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:01:47.942365 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:47.942048 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:01:48.032628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:48.018346 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:01:48.046720 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:48.038647 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:01:48.052704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:01:48.052673 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:01:57.781495 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:01:57.781459 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:02:05.907632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:05.906466 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:02:05.964629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:05.960745 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:02:06.044758 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:06.044699 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:02:06.046224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:06.046189 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 19:02:06.058860 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:06.058813 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 19:02:06.189804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:06.189698 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:02:16.073493 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.073452 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5"] Oct 30 19:02:16.075519 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.074494 5959 cpu_manager.go:414] "containerMap: removing container" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:02:16.075519 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.074517 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85147ab0-84fb-4d83-beeb-2a8e0325130c" containerName="cleanup" Oct 30 19:02:16.075519 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.074535 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:02:16.075519 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.074543 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd79fee-1bed-47a0-aa08-7a8b559318a8" containerName="cleanup" Oct 30 19:02:16.079183 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.079055 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:16.093289 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.093255 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5"] Oct 30 19:02:16.154852 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.154806 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-baa504\"" Oct 30 19:02:16.155459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.155436 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-baa504-dockercfg-pcngq\"" Oct 30 19:02:16.248658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.238947 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5\" (UID: \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:16.248658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.239238 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29d58\" (UniqueName: \"kubernetes.io/projected/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-kube-api-access-29d58\") pod \"payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5\" (UID: \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:16.342008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.341651 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29d58\" (UniqueName: \"kubernetes.io/projected/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-kube-api-access-29d58\") pod \"payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5\" (UID: \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:16.342008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.341800 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5\" (UID: \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:16.343400 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.342829 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5\" (UID: \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:16.372631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.372136 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29d58\" (UniqueName: \"kubernetes.io/projected/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-kube-api-access-29d58\") pod \"payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5\" (UID: \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:16.411395 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.411295 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:16.713844 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:16.713807 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5"] Oct 30 19:02:16.718814 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:02:16.718771 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e95592_3e89_49e5_a892_2cd8b5ddcb1f.slice/crio-f4071023c48cb0696885e7e71ed7bd0ab867095c381fa6b49a788517d7a9a4ad WatchSource:0}: Error finding container f4071023c48cb0696885e7e71ed7bd0ab867095c381fa6b49a788517d7a9a4ad: Status 404 returned error can't find the container with id f4071023c48cb0696885e7e71ed7bd0ab867095c381fa6b49a788517d7a9a4ad Oct 30 19:02:17.301926 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:17.301892 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" event={"ID":"45e95592-3e89-49e5-a892-2cd8b5ddcb1f","Type":"ContainerStarted","Data":"762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8"} Oct 30 19:02:17.302567 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:17.301953 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" event={"ID":"45e95592-3e89-49e5-a892-2cd8b5ddcb1f","Type":"ContainerStarted","Data":"f4071023c48cb0696885e7e71ed7bd0ab867095c381fa6b49a788517d7a9a4ad"} Oct 30 19:02:17.302567 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:17.302028 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:17.324651 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:17.324575 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" podStartSLOduration=1.324557905 podStartE2EDuration="1.324557905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:17.319173735 +0000 UTC m=+1517.647585119" watchObservedRunningTime="2026-10-30 19:02:17.324557905 +0000 UTC m=+1517.652969333" Oct 30 19:02:21.638759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.638725 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v"] Oct 30 19:02:21.643009 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.642987 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:21.649778 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.649754 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ef6be7\"" Oct 30 19:02:21.650882 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.650858 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ef6be7-dockercfg-k9cm2\"" Oct 30 19:02:21.655022 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.655000 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v"] Oct 30 19:02:21.667773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.667743 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6"] Oct 30 19:02:21.672004 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.671982 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:21.680841 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.680816 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6"] Oct 30 19:02:21.749459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.748941 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v\" (UID: \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:21.749459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.749003 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb79l\" (UniqueName: \"kubernetes.io/projected/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-kube-api-access-tb79l\") pod \"payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v\" (UID: \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:21.749459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.749047 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6\" (UID: \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:21.749459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.749091 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfjr\" (UniqueName: \"kubernetes.io/projected/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-kube-api-access-vpfjr\") pod \"payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6\" (UID: \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:21.850756 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.850721 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v\" (UID: \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:21.850956 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.850765 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb79l\" (UniqueName: \"kubernetes.io/projected/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-kube-api-access-tb79l\") pod \"payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v\" (UID: \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:21.850956 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.850903 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6\" (UID: \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:21.850956 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.850951 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfjr\" (UniqueName: \"kubernetes.io/projected/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-kube-api-access-vpfjr\") pod \"payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6\" (UID: \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:21.851539 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.851516 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v\" (UID: \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:21.851715 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.851699 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6\" (UID: \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:21.858877 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.858846 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfjr\" (UniqueName: \"kubernetes.io/projected/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-kube-api-access-vpfjr\") pod \"payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6\" (UID: \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:21.859013 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.858942 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb79l\" (UniqueName: \"kubernetes.io/projected/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-kube-api-access-tb79l\") pod \"payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v\" (UID: \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:21.961072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.960980 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:21.990506 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:21.990470 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:22.257573 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:22.257549 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v"] Oct 30 19:02:22.259738 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:02:22.259706 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7d1645_e7a7_433a_a39f_8d53f47d01ef.slice/crio-f4d8bb2f4299c3c40aa63548d859a374803d9817f39068f4bbf62c37091d86eb WatchSource:0}: Error finding container f4d8bb2f4299c3c40aa63548d859a374803d9817f39068f4bbf62c37091d86eb: Status 404 returned error can't find the container with id f4d8bb2f4299c3c40aa63548d859a374803d9817f39068f4bbf62c37091d86eb Oct 30 19:02:22.310968 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:22.310937 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6"] Oct 30 19:02:22.313303 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:02:22.313276 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2581e0a_b5b3_4df8_8b36_7f9b5232addc.slice/crio-03f7e4c9b6fb97f9e6656385eed9ba9d96f2ed2f817151d71e9c6e4974179875 WatchSource:0}: Error finding container 03f7e4c9b6fb97f9e6656385eed9ba9d96f2ed2f817151d71e9c6e4974179875: Status 404 returned error can't find the container with id 03f7e4c9b6fb97f9e6656385eed9ba9d96f2ed2f817151d71e9c6e4974179875 Oct 30 19:02:22.341543 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:22.341509 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" event={"ID":"f2581e0a-b5b3-4df8-8b36-7f9b5232addc","Type":"ContainerStarted","Data":"03f7e4c9b6fb97f9e6656385eed9ba9d96f2ed2f817151d71e9c6e4974179875"} Oct 30 19:02:22.343069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:22.343045 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" event={"ID":"cb7d1645-e7a7-433a-a39f-8d53f47d01ef","Type":"ContainerStarted","Data":"926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140"} Oct 30 19:02:22.343191 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:22.343092 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" event={"ID":"cb7d1645-e7a7-433a-a39f-8d53f47d01ef","Type":"ContainerStarted","Data":"f4d8bb2f4299c3c40aa63548d859a374803d9817f39068f4bbf62c37091d86eb"} Oct 30 19:02:22.343191 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:22.343160 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:22.361222 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:22.361170 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" podStartSLOduration=1.361157777 podStartE2EDuration="1.361157777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:22.359068518 +0000 UTC m=+1522.687479914" watchObservedRunningTime="2026-10-30 19:02:22.361157777 +0000 UTC m=+1522.689569160" Oct 30 19:02:23.351149 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:23.351107 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" event={"ID":"f2581e0a-b5b3-4df8-8b36-7f9b5232addc","Type":"ContainerStarted","Data":"dcce2ff4b1ba68c4e146a3c3506a3e40ae67262aabfd3c2dbbff95b37adb21c2"} Oct 30 19:02:23.351662 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:23.351218 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:23.369585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:23.369536 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" podStartSLOduration=2.36952125 podStartE2EDuration="2.36952125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:23.365966155 +0000 UTC m=+1523.694377538" watchObservedRunningTime="2026-10-30 19:02:23.36952125 +0000 UTC m=+1523.697932632" Oct 30 19:02:27.780524 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:02:27.780476 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:02:28.313655 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:28.313621 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:02:29.862811 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:29.862778 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:02:30.802736 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.802683 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz"] Oct 30 19:02:30.809311 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.809281 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz"] Oct 30 19:02:30.811106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.811080 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.821026 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.821003 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ef6be7-serving-cert\"" Oct 30 19:02:30.839999 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.839961 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrx5m\" (UniqueName: \"kubernetes.io/projected/0aa10346-b095-4bcb-83c9-b09741af930d-kube-api-access-vrx5m\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.840281 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.840257 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.840512 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.840486 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa10346-b095-4bcb-83c9-b09741af930d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.840985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.840953 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.942885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.942848 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.943403 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.942943 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa10346-b095-4bcb-83c9-b09741af930d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.943403 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.943118 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.943403 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.943290 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrx5m\" (UniqueName: \"kubernetes.io/projected/0aa10346-b095-4bcb-83c9-b09741af930d-kube-api-access-vrx5m\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.943574 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:02:30.943527 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-serving-cert: secret "maas-api-e2e-shared-b-w2-ef6be7-serving-cert" not found Oct 30 19:02:30.943667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.943570 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa10346-b095-4bcb-83c9-b09741af930d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.943667 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:02:30.943594 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls podName:0aa10346-b095-4bcb-83c9-b09741af930d nodeName:}" failed. No retries permitted until 2026-10-30 19:02:31.443574149 +0000 UTC m=+1531.771985515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" (UID: "0aa10346-b095-4bcb-83c9-b09741af930d") : secret "maas-api-e2e-shared-b-w2-ef6be7-serving-cert" not found Oct 30 19:02:30.945903 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.945876 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:30.954771 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:30.954707 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrx5m\" (UniqueName: \"kubernetes.io/projected/0aa10346-b095-4bcb-83c9-b09741af930d-kube-api-access-vrx5m\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:31.457014 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:31.456970 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:31.457381 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:02:31.457361 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-serving-cert: secret "maas-api-e2e-shared-b-w2-ef6be7-serving-cert" not found Oct 30 19:02:31.457485 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:02:31.457432 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls podName:0aa10346-b095-4bcb-83c9-b09741af930d nodeName:}" failed. No retries permitted until 2026-10-30 19:02:32.457412809 +0000 UTC m=+1532.785824171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" (UID: "0aa10346-b095-4bcb-83c9-b09741af930d") : secret "maas-api-e2e-shared-b-w2-ef6be7-serving-cert" not found Oct 30 19:02:32.486779 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:32.486739 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:32.497308 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:32.497270 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:32.647927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:32.647893 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:33.194731 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:33.193329 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz"] Oct 30 19:02:33.194945 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:02:33.194918 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa10346_b095_4bcb_83c9_b09741af930d.slice/crio-1e051cb944aa5cbe98754bb8e266a43d2ffcb2f190aee0d3ff08116aad7dbb40 WatchSource:0}: Error finding container 1e051cb944aa5cbe98754bb8e266a43d2ffcb2f190aee0d3ff08116aad7dbb40: Status 404 returned error can't find the container with id 1e051cb944aa5cbe98754bb8e266a43d2ffcb2f190aee0d3ff08116aad7dbb40 Oct 30 19:02:33.197124 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:33.197101 5959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:02:33.354428 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:33.354340 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:02:33.446151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:33.446111 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" event={"ID":"0aa10346-b095-4bcb-83c9-b09741af930d","Type":"ContainerStarted","Data":"1e051cb944aa5cbe98754bb8e266a43d2ffcb2f190aee0d3ff08116aad7dbb40"} Oct 30 19:02:34.386630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:34.372883 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:02:34.551791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:34.551436 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" event={"ID":"0aa10346-b095-4bcb-83c9-b09741af930d","Type":"ContainerStarted","Data":"ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715"} Oct 30 19:02:34.554018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:34.552573 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:34.577898 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:34.577835 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" podStartSLOduration=4.081258156 podStartE2EDuration="4.577817841s" totalImagesPullingTime="496.559685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:34.573520112 +0000 UTC m=+1534.901931498" watchObservedRunningTime="2026-10-30 19:02:34.577817841 +0000 UTC m=+1534.906229227" Oct 30 19:02:34.693028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:34.692935 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs"] Oct 30 19:02:34.693216 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:34.693179 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" podUID="f634b8e0-7f6c-491d-803a-ada7f7b0627f" containerName="maas-api" containerID="cri-o://5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623" gracePeriod=30 Oct 30 19:02:35.012524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.012486 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 19:02:35.154638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.144223 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-maas-api-tls\") pod \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " Oct 30 19:02:35.154638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.144479 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f634b8e0-7f6c-491d-803a-ada7f7b0627f-kube-api-access-wjxc5\" (UniqueName: \"kubernetes.io/projected/f634b8e0-7f6c-491d-803a-ada7f7b0627f-kube-api-access-wjxc5\") pod \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " Oct 30 19:02:35.154638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.144552 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f634b8e0-7f6c-491d-803a-ada7f7b0627f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f634b8e0-7f6c-491d-803a-ada7f7b0627f-openshift-service-ca\") pod \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " Oct 30 19:02:35.154638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.144593 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-metrics-certs\") pod \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\" (UID: \"f634b8e0-7f6c-491d-803a-ada7f7b0627f\") " Oct 30 19:02:35.154638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.149111 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-metrics-certs" pod "f634b8e0-7f6c-491d-803a-ada7f7b0627f" (UID: "f634b8e0-7f6c-491d-803a-ada7f7b0627f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:35.172976 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.172932 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-maas-api-tls" pod "f634b8e0-7f6c-491d-803a-ada7f7b0627f" (UID: "f634b8e0-7f6c-491d-803a-ada7f7b0627f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:35.177594 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.177553 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f634b8e0-7f6c-491d-803a-ada7f7b0627f-kube-api-access-wjxc5" pod "f634b8e0-7f6c-491d-803a-ada7f7b0627f" (UID: "f634b8e0-7f6c-491d-803a-ada7f7b0627f"). InnerVolumeSpecName "kube-api-access-wjxc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:35.177952 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.177931 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f634b8e0-7f6c-491d-803a-ada7f7b0627f-openshift-service-ca" pod "f634b8e0-7f6c-491d-803a-ada7f7b0627f" (UID: "f634b8e0-7f6c-491d-803a-ada7f7b0627f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:35.247091 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.246997 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f634b8e0-7f6c-491d-803a-ada7f7b0627f-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:02:35.247091 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.247028 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:02:35.247091 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.247041 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f634b8e0-7f6c-491d-803a-ada7f7b0627f-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:02:35.247091 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.247053 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjxc5\" (UniqueName: \"kubernetes.io/projected/f634b8e0-7f6c-491d-803a-ada7f7b0627f-kube-api-access-wjxc5\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:02:35.559863 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.559770 5959 generic.go:356] "Generic (PLEG): container finished" podID="f634b8e0-7f6c-491d-803a-ada7f7b0627f" containerID="5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623" exitCode=0 Oct 30 19:02:35.559863 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.559829 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" Oct 30 19:02:35.560386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.559865 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" event={"ID":"f634b8e0-7f6c-491d-803a-ada7f7b0627f","Type":"ContainerDied","Data":"5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623"} Oct 30 19:02:35.560386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.559926 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs" event={"ID":"f634b8e0-7f6c-491d-803a-ada7f7b0627f","Type":"ContainerDied","Data":"962ef2cb63a1b7e69737b6d20dd8f43aefc292925a9b9352c58f81ee9a9cc1ab"} Oct 30 19:02:35.560386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.559953 5959 scope.go:122] "RemoveContainer" containerID="5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623" Oct 30 19:02:35.571462 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.571444 5959 scope.go:122] "RemoveContainer" containerID="5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623" Oct 30 19:02:35.571789 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:02:35.571769 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623\": container with ID starting with 5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623 not found: ID does not exist" containerID="5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623" Oct 30 19:02:35.571845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.571805 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623"} err="failed to get container status \"5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623\": rpc error: code = NotFound desc = could not find container \"5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623\": container with ID starting with 5f6a0977f602537aeb5f5adad93a8077931e5f0f345865365bafa8899c46f623 not found: ID does not exist" Oct 30 19:02:35.583056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.583028 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs"] Oct 30 19:02:35.587929 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.587902 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a014d1-6ccc78c7-9ghvs"] Oct 30 19:02:35.797871 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:35.797840 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f634b8e0-7f6c-491d-803a-ada7f7b0627f" path="/var/lib/kubelet/pods/f634b8e0-7f6c-491d-803a-ada7f7b0627f/volumes" Oct 30 19:02:41.598285 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:41.598208 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:02:42.257790 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.257757 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc"] Oct 30 19:02:42.258532 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.258502 5959 cpu_manager.go:414] "containerMap: removing container" podUID="f634b8e0-7f6c-491d-803a-ada7f7b0627f" containerName="maas-api" Oct 30 19:02:42.258532 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.258525 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f634b8e0-7f6c-491d-803a-ada7f7b0627f" containerName="maas-api" Oct 30 19:02:42.258746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.258737 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f634b8e0-7f6c-491d-803a-ada7f7b0627f" containerName="maas-api" Oct 30 19:02:42.264318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.264292 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:42.321628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.303015 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc"] Oct 30 19:02:42.346905 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.346299 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2d1c71-dockercfg-7tlhj\"" Oct 30 19:02:42.370662 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.367891 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2d1c71\"" Oct 30 19:02:42.404627 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.404394 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc\" (UID: \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:42.404627 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.404595 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfzd\" (UniqueName: \"kubernetes.io/projected/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-kube-api-access-pvfzd\") pod \"payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc\" (UID: \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:42.506355 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.506315 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfzd\" (UniqueName: \"kubernetes.io/projected/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-kube-api-access-pvfzd\") pod \"payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc\" (UID: \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:42.506546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.506467 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc\" (UID: \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:42.507213 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.507191 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc\" (UID: \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:42.515083 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.515013 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfzd\" (UniqueName: \"kubernetes.io/projected/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-kube-api-access-pvfzd\") pod \"payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc\" (UID: \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:42.649001 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.648970 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:42.932948 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:42.932922 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc"] Oct 30 19:02:42.934851 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:02:42.934820 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fef6d6c_ffa6_4ce8_a406_e4265da4f9d7.slice/crio-962958978fd28341e6cec3a7046004b5498efc3c9470bfd4a9aa78265f1af478 WatchSource:0}: Error finding container 962958978fd28341e6cec3a7046004b5498efc3c9470bfd4a9aa78265f1af478: Status 404 returned error can't find the container with id 962958978fd28341e6cec3a7046004b5498efc3c9470bfd4a9aa78265f1af478 Oct 30 19:02:43.642365 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:43.642334 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" event={"ID":"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7","Type":"ContainerStarted","Data":"94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479"} Oct 30 19:02:43.642569 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:43.642403 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" event={"ID":"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7","Type":"ContainerStarted","Data":"962958978fd28341e6cec3a7046004b5498efc3c9470bfd4a9aa78265f1af478"} Oct 30 19:02:43.642569 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:43.642432 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:43.661519 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:43.661464 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" podStartSLOduration=1.6614482590000001 podStartE2EDuration="1.661448259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:43.658993833 +0000 UTC m=+1543.987405216" watchObservedRunningTime="2026-10-30 19:02:43.661448259 +0000 UTC m=+1543.989859643" Oct 30 19:02:50.811297 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:50.811263 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp"] Oct 30 19:02:50.815327 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:50.815310 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:50.822069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:50.822047 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-68b466\"" Oct 30 19:02:50.826533 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:50.826495 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp"] Oct 30 19:02:50.987630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:50.984513 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttlp\" (UniqueName: \"kubernetes.io/projected/21ebf898-38d1-4786-8ebc-d5e53a52961f-kube-api-access-zttlp\") pod \"payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp\" (UID: \"21ebf898-38d1-4786-8ebc-d5e53a52961f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:50.987630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:50.984697 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ebf898-38d1-4786-8ebc-d5e53a52961f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp\" (UID: \"21ebf898-38d1-4786-8ebc-d5e53a52961f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:51.099008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.098913 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttlp\" (UniqueName: \"kubernetes.io/projected/21ebf898-38d1-4786-8ebc-d5e53a52961f-kube-api-access-zttlp\") pod \"payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp\" (UID: \"21ebf898-38d1-4786-8ebc-d5e53a52961f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:51.099199 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.099046 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ebf898-38d1-4786-8ebc-d5e53a52961f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp\" (UID: \"21ebf898-38d1-4786-8ebc-d5e53a52961f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:51.099906 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.099879 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ebf898-38d1-4786-8ebc-d5e53a52961f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp\" (UID: \"21ebf898-38d1-4786-8ebc-d5e53a52961f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:51.113735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.113705 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttlp\" (UniqueName: \"kubernetes.io/projected/21ebf898-38d1-4786-8ebc-d5e53a52961f-kube-api-access-zttlp\") pod \"payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp\" (UID: \"21ebf898-38d1-4786-8ebc-d5e53a52961f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:51.130480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.130456 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:51.419681 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.419657 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp"] Oct 30 19:02:51.421555 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:02:51.421522 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ebf898_38d1_4786_8ebc_d5e53a52961f.slice/crio-182ee51656881b7212941e051ac45183e65a9278dfd9d57728c67db955d4af6c WatchSource:0}: Error finding container 182ee51656881b7212941e051ac45183e65a9278dfd9d57728c67db955d4af6c: Status 404 returned error can't find the container with id 182ee51656881b7212941e051ac45183e65a9278dfd9d57728c67db955d4af6c Oct 30 19:02:51.739109 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.739016 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" event={"ID":"21ebf898-38d1-4786-8ebc-d5e53a52961f","Type":"ContainerStarted","Data":"7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214"} Oct 30 19:02:51.739109 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.739080 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" event={"ID":"21ebf898-38d1-4786-8ebc-d5e53a52961f","Type":"ContainerStarted","Data":"182ee51656881b7212941e051ac45183e65a9278dfd9d57728c67db955d4af6c"} Oct 30 19:02:51.739310 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.739139 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:02:51.759512 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:51.759453 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" podStartSLOduration=1.7594343220000002 podStartE2EDuration="1.759434322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:51.756894401 +0000 UTC m=+1552.085305784" watchObservedRunningTime="2026-10-30 19:02:51.759434322 +0000 UTC m=+1552.087845707" Oct 30 19:02:52.469127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.469092 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb"] Oct 30 19:02:52.474368 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.474343 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.484203 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.483318 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb"] Oct 30 19:02:52.484203 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.483805 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-2d1c71-serving-cert\"" Oct 30 19:02:52.622149 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.622106 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8mc\" (UniqueName: \"kubernetes.io/projected/122408a8-f58c-46bf-b236-5beaf8afd104-kube-api-access-fl8mc\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.622345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.622243 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/122408a8-f58c-46bf-b236-5beaf8afd104-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.622416 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.622351 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.622503 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.622488 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.723670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.723547 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8mc\" (UniqueName: \"kubernetes.io/projected/122408a8-f58c-46bf-b236-5beaf8afd104-kube-api-access-fl8mc\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.723855 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.723672 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/122408a8-f58c-46bf-b236-5beaf8afd104-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.723855 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.723799 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.723989 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.723967 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.724750 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.724724 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/122408a8-f58c-46bf-b236-5beaf8afd104-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.726774 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.726751 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.728120 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.728100 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.733065 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.732998 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8mc\" (UniqueName: \"kubernetes.io/projected/122408a8-f58c-46bf-b236-5beaf8afd104-kube-api-access-fl8mc\") pod \"maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:52.801404 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:52.801378 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:53.099810 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:53.099785 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb"] Oct 30 19:02:53.103070 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:02:53.103042 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122408a8_f58c_46bf_b236_5beaf8afd104.slice/crio-dbc0d9a32eff92c42066aa68b1ac14e01222293cdbd2bdb20006ee785dfcb095 WatchSource:0}: Error finding container dbc0d9a32eff92c42066aa68b1ac14e01222293cdbd2bdb20006ee785dfcb095: Status 404 returned error can't find the container with id dbc0d9a32eff92c42066aa68b1ac14e01222293cdbd2bdb20006ee785dfcb095 Oct 30 19:02:53.755080 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:53.754987 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" event={"ID":"122408a8-f58c-46bf-b236-5beaf8afd104","Type":"ContainerStarted","Data":"1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0"} Oct 30 19:02:53.755080 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:53.755046 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" event={"ID":"122408a8-f58c-46bf-b236-5beaf8afd104","Type":"ContainerStarted","Data":"dbc0d9a32eff92c42066aa68b1ac14e01222293cdbd2bdb20006ee785dfcb095"} Oct 30 19:02:53.755080 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:53.755079 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:02:53.776773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:53.776726 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" podStartSLOduration=1.382491739 podStartE2EDuration="1.776708304s" totalImagesPullingTime="394.216565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:53.774184814 +0000 UTC m=+1554.102596198" watchObservedRunningTime="2026-10-30 19:02:53.776708304 +0000 UTC m=+1554.105119690" Oct 30 19:02:54.662594 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:54.661098 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:02:55.302955 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:55.302920 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:02:57.787232 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:02:57.787198 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:02:59.768943 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:02:59.768912 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:03:01.082824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:01.082783 5959 scope.go:122] "RemoveContainer" containerID="5dd5a861cb53b24cf7412bf809349e5436b28ef06633d6fcd43380d919be86c3" Oct 30 19:03:02.749638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:02.749595 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:03:10.571517 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.571480 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw"] Oct 30 19:03:10.577993 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.577967 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:10.587403 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.587373 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw"] Oct 30 19:03:10.639637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.639574 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw\" (UID: \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:10.639968 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.639947 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75fg\" (UniqueName: \"kubernetes.io/projected/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-kube-api-access-g75fg\") pod \"payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw\" (UID: \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:10.670482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.670453 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-e9ada0\"" Oct 30 19:03:10.670724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.670585 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-e9ada0-dockercfg-whzgm\"" Oct 30 19:03:10.741846 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.741802 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw\" (UID: \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:10.742292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.742267 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75fg\" (UniqueName: \"kubernetes.io/projected/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-kube-api-access-g75fg\") pod \"payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw\" (UID: \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:10.743667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.743639 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw\" (UID: \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:10.753663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.751963 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75fg\" (UniqueName: \"kubernetes.io/projected/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-kube-api-access-g75fg\") pod \"payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw\" (UID: \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:10.901185 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:10.901148 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:11.192936 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:11.192903 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw"] Oct 30 19:03:11.195063 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:03:11.195019 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e9ccca_f161_44a9_a52e_627fc8fc5cf2.slice/crio-447d015e867e89f03fe179278f5060857abdefe48574b610f4be88a98025ec3d WatchSource:0}: Error finding container 447d015e867e89f03fe179278f5060857abdefe48574b610f4be88a98025ec3d: Status 404 returned error can't find the container with id 447d015e867e89f03fe179278f5060857abdefe48574b610f4be88a98025ec3d Oct 30 19:03:11.903081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:11.903042 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" event={"ID":"67e9ccca-f161-44a9-a52e-627fc8fc5cf2","Type":"ContainerStarted","Data":"bca1721a36f6de4b7119202a56e54c7ec3d53d0ff52161b0d852c58cea5335c7"} Oct 30 19:03:11.903582 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:11.903102 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" event={"ID":"67e9ccca-f161-44a9-a52e-627fc8fc5cf2","Type":"ContainerStarted","Data":"447d015e867e89f03fe179278f5060857abdefe48574b610f4be88a98025ec3d"} Oct 30 19:03:11.903582 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:11.903130 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:11.925906 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:11.925850 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" podStartSLOduration=1.925831616 podStartE2EDuration="1.925831616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:11.924523283 +0000 UTC m=+1572.252934668" watchObservedRunningTime="2026-10-30 19:03:11.925831616 +0000 UTC m=+1572.254242999" Oct 30 19:03:21.645559 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.645525 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m"] Oct 30 19:03:21.651468 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.651437 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.658338 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.658306 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-e9ada0-serving-cert\"" Oct 30 19:03:21.661105 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.661076 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m"] Oct 30 19:03:21.683962 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.683926 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.684185 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.684122 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708a6acc-01cf-43f2-9473-d71b91aa395a-openshift-service-ca\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.684337 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.684312 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-metrics-certs\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.684476 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.684347 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngpl\" (UniqueName: \"kubernetes.io/projected/708a6acc-01cf-43f2-9473-d71b91aa395a-kube-api-access-pngpl\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.786129 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.786084 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-metrics-certs\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.786436 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.786405 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngpl\" (UniqueName: \"kubernetes.io/projected/708a6acc-01cf-43f2-9473-d71b91aa395a-kube-api-access-pngpl\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.786636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.786620 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.786761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.786744 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708a6acc-01cf-43f2-9473-d71b91aa395a-openshift-service-ca\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.787211 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:03:21.786814 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-serving-cert: secret "maas-api-e2e-explicit-e9ada0-serving-cert" not found Oct 30 19:03:21.787211 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:03:21.787033 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls podName:708a6acc-01cf-43f2-9473-d71b91aa395a nodeName:}" failed. No retries permitted until 2026-10-30 19:03:22.287001666 +0000 UTC m=+1582.615413031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls") pod "maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" (UID: "708a6acc-01cf-43f2-9473-d71b91aa395a") : secret "maas-api-e2e-explicit-e9ada0-serving-cert" not found Oct 30 19:03:21.787570 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.787543 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708a6acc-01cf-43f2-9473-d71b91aa395a-openshift-service-ca\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.802323 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.789552 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-metrics-certs\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:21.808750 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:21.805555 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngpl\" (UniqueName: \"kubernetes.io/projected/708a6acc-01cf-43f2-9473-d71b91aa395a-kube-api-access-pngpl\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:22.295998 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:22.295963 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:22.298666 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:22.298636 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls\") pod \"maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:22.582161 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:22.581839 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:22.928410 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:22.928265 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:03:22.932104 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:03:22.932057 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708a6acc_01cf_43f2_9473_d71b91aa395a.slice/crio-984f52895544a377106453202c10bcf769082a31aad95fd61531e8ae1f277f42 WatchSource:0}: Error finding container 984f52895544a377106453202c10bcf769082a31aad95fd61531e8ae1f277f42: Status 404 returned error can't find the container with id 984f52895544a377106453202c10bcf769082a31aad95fd61531e8ae1f277f42 Oct 30 19:03:22.933334 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:22.933308 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m"] Oct 30 19:03:23.010577 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:23.010537 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" event={"ID":"708a6acc-01cf-43f2-9473-d71b91aa395a","Type":"ContainerStarted","Data":"984f52895544a377106453202c10bcf769082a31aad95fd61531e8ae1f277f42"} Oct 30 19:03:24.020393 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:24.020351 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" event={"ID":"708a6acc-01cf-43f2-9473-d71b91aa395a","Type":"ContainerStarted","Data":"24e72fedf1d30929bc5f70e9ba9a9248a29cf431be7a175b38c5bd48b8f617ce"} Oct 30 19:03:24.020949 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:24.020566 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:24.046590 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:24.045350 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" podStartSLOduration=2.50767132 podStartE2EDuration="3.045327357s" totalImagesPullingTime="537.656037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:24.040755581 +0000 UTC m=+1584.369166966" watchObservedRunningTime="2026-10-30 19:03:24.045327357 +0000 UTC m=+1584.373738747" Oct 30 19:03:27.781599 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:03:27.781563 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:03:30.034387 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:30.034354 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:03:36.450785 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.450753 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5"] Oct 30 19:03:36.455864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.455841 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.466817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.466774 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2d1c71\"/\"openshift-service-ca.crt\"" Oct 30 19:03:36.467034 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.466957 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2d1c71\"/\"kube-root-ca.crt\"" Oct 30 19:03:36.467170 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.467150 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2d1c71\"/\"auth-test-model-2d1c71-kserve-self-signed-certs\"" Oct 30 19:03:36.467263 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.467239 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2d1c71\"/\"default-dockercfg-fwx9k\"" Oct 30 19:03:36.469354 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.469325 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5"] Oct 30 19:03:36.636478 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.636434 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2276f-8831-4420-8355-44363744c018-tls-certs\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.636686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.636505 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-dshm\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.636735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.636709 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-model-cache\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.636785 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.636764 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-home\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.636986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.636971 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-tmp-dir\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.637077 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.637059 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjd8z\" (UniqueName: \"kubernetes.io/projected/73b2276f-8831-4420-8355-44363744c018-kube-api-access-jjd8z\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.741688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.741569 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-tmp-dir\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.741688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.741683 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjd8z\" (UniqueName: \"kubernetes.io/projected/73b2276f-8831-4420-8355-44363744c018-kube-api-access-jjd8z\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.741972 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.741768 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2276f-8831-4420-8355-44363744c018-tls-certs\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.741972 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.741811 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-dshm\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.741972 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.741878 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-model-cache\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.741972 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.741919 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-home\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.747701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.742442 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-home\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.747701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.742713 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-tmp-dir\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.750214 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.750180 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-model-cache\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.754506 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.754475 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjd8z\" (UniqueName: \"kubernetes.io/projected/73b2276f-8831-4420-8355-44363744c018-kube-api-access-jjd8z\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.774630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.774213 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2276f-8831-4420-8355-44363744c018-tls-certs\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:36.778638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:36.777105 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-dshm\") pod \"auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:37.073384 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:37.073293 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:37.357048 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:37.357017 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5"] Oct 30 19:03:37.357880 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:03:37.357854 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73b2276f_8831_4420_8355_44363744c018.slice/crio-92371c096b823ed02ee0bdbe3ed5fb1df51cc1f9a28cfc8fe10585af9c7bcbf3 WatchSource:0}: Error finding container 92371c096b823ed02ee0bdbe3ed5fb1df51cc1f9a28cfc8fe10585af9c7bcbf3: Status 404 returned error can't find the container with id 92371c096b823ed02ee0bdbe3ed5fb1df51cc1f9a28cfc8fe10585af9c7bcbf3 Oct 30 19:03:38.140454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:38.140424 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" event={"ID":"73b2276f-8831-4420-8355-44363744c018","Type":"ContainerStarted","Data":"7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc"} Oct 30 19:03:38.140454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:38.140462 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" event={"ID":"73b2276f-8831-4420-8355-44363744c018","Type":"ContainerStarted","Data":"92371c096b823ed02ee0bdbe3ed5fb1df51cc1f9a28cfc8fe10585af9c7bcbf3"} Oct 30 19:03:38.160552 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:38.160496 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" podStartSLOduration=2.160480939 podStartE2EDuration="2.160480939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:38.157953611 +0000 UTC m=+1598.486364995" watchObservedRunningTime="2026-10-30 19:03:38.160480939 +0000 UTC m=+1598.488892323" Oct 30 19:03:39.073509 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:39.073475 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:42.642495 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:42.642459 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:03:43.363690 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:43.363647 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5"] Oct 30 19:03:43.363915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:43.363893 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" containerID="cri-o://712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9" gracePeriod=30 Oct 30 19:03:47.073804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:47.073771 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:47.087172 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:47.087144 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:47.245188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:47.245142 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:03:48.687822 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.687795 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 19:03:48.759587 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.759545 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-data\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.759805 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.759657 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-socket\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.759805 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.759759 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-credential-socket\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.759941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.759925 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-envoy\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.759941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.759923 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-data" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:48.760046 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.759948 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-socket" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:48.760046 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.759964 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-kube-api-access-pfm89\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-kube-api-access-pfm89\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.760046 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760036 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-certs\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.760211 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760097 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-podinfo\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.760211 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760157 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-token\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-token\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.760211 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760167 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-credential-socket" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:48.760370 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760223 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istiod-ca-cert\") pod \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\" (UID: \"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a\") " Oct 30 19:03:48.760370 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760353 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-certs" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:48.760889 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760864 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:48.761016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760892 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:48.761016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760909 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:48.761016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.760924 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:48.761319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.761193 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istiod-ca-cert" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:48.763427 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.763405 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-podinfo" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:03:48.763517 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.763492 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-token" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:48.763560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.763510 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-kube-api-access-pfm89" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "kube-api-access-pfm89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:48.763769 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.763745 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-envoy" pod "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" (UID: "05c7c1c0-16bf-4637-ae86-99e9a19dfd8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:48.861967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.861927 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:48.861967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.861962 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:48.861967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.861973 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfm89\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-kube-api-access-pfm89\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:48.862218 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.861984 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:48.862218 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:48.861993 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:03:49.236403 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.236348 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg"] Oct 30 19:03:49.237508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.237477 5959 cpu_manager.go:414] "containerMap: removing container" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" Oct 30 19:03:49.237508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.237507 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" Oct 30 19:03:49.237781 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.237762 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerName="istio-proxy" Oct 30 19:03:49.245236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.243232 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.246648 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.246580 5959 generic.go:356] "Generic (PLEG): container finished" podID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" containerID="712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9" exitCode=0 Oct 30 19:03:49.246791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.246675 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" Oct 30 19:03:49.246791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.246759 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" event={"ID":"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a","Type":"ContainerDied","Data":"712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9"} Oct 30 19:03:49.246904 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.246799 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5" event={"ID":"05c7c1c0-16bf-4637-ae86-99e9a19dfd8a","Type":"ContainerDied","Data":"1983897576c3f2dcbb479c3fbccae2a4442623afd8621248c3964ee2c27a80fb"} Oct 30 19:03:49.246904 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.246830 5959 scope.go:122] "RemoveContainer" containerID="712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9" Oct 30 19:03:49.251155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.250531 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e9ada0\"/\"kube-root-ca.crt\"" Oct 30 19:03:49.251155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.250855 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg"] Oct 30 19:03:49.251499 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.251436 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e9ada0\"/\"default-dockercfg-7gbqb\"" Oct 30 19:03:49.251585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.251539 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e9ada0\"/\"explicit-model-e9ada0-kserve-self-signed-certs\"" Oct 30 19:03:49.252863 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.252843 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e9ada0\"/\"openshift-service-ca.crt\"" Oct 30 19:03:49.270337 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.270308 5959 scope.go:122] "RemoveContainer" containerID="712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9" Oct 30 19:03:49.270842 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:03:49.270818 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9\": container with ID starting with 712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9 not found: ID does not exist" containerID="712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9" Oct 30 19:03:49.270939 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.270853 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9"} err="failed to get container status \"712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9\": rpc error: code = NotFound desc = could not find container \"712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9\": container with ID starting with 712f2b2193b875e310cb404ed555443eb382a588fd83ab0105b257a0806ef5a9 not found: ID does not exist" Oct 30 19:03:49.289967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.289922 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5"] Oct 30 19:03:49.293281 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.293250 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-8hjs5"] Oct 30 19:03:49.369523 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.369483 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-model-cache\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.369741 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.369565 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tmp-dir\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.369741 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.369712 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tls-certs\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.369862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.369762 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-dshm\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.369989 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.369966 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8w8\" (UniqueName: \"kubernetes.io/projected/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-kube-api-access-xl8w8\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.370121 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.370095 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-home\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.471676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.471636 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8w8\" (UniqueName: \"kubernetes.io/projected/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-kube-api-access-xl8w8\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.472029 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.472008 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-home\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.472236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.472222 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-model-cache\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.472380 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.472366 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tmp-dir\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.472496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.472481 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tls-certs\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.472702 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.472681 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-dshm\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.481627 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.476074 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tmp-dir\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.492456 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.492308 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8w8\" (UniqueName: \"kubernetes.io/projected/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-kube-api-access-xl8w8\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.493070 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.493043 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-model-cache\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.493364 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.493297 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-home\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.507632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.495134 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tls-certs\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.507632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.495290 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-dshm\") pod \"explicit-model-e9ada0-kserve-5dd59d7f75-kpftg\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.574691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.574651 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:49.798948 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.798870 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c7c1c0-16bf-4637-ae86-99e9a19dfd8a" path="/var/lib/kubelet/pods/05c7c1c0-16bf-4637-ae86-99e9a19dfd8a/volumes" Oct 30 19:03:49.880528 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:49.880496 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg"] Oct 30 19:03:49.881928 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:03:49.881899 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e5f24dd_c99c_44aa_99a6_5a5cb0c2c51b.slice/crio-ee95cfaa95054a92655f9f8f2cef095926cb1328da5d8fb77916716367abaaf5 WatchSource:0}: Error finding container ee95cfaa95054a92655f9f8f2cef095926cb1328da5d8fb77916716367abaaf5: Status 404 returned error can't find the container with id ee95cfaa95054a92655f9f8f2cef095926cb1328da5d8fb77916716367abaaf5 Oct 30 19:03:50.255914 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.255867 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" event={"ID":"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b","Type":"ContainerStarted","Data":"1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98"} Oct 30 19:03:50.256136 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.255922 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" event={"ID":"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b","Type":"ContainerStarted","Data":"ee95cfaa95054a92655f9f8f2cef095926cb1328da5d8fb77916716367abaaf5"} Oct 30 19:03:50.280397 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.280340 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" podStartSLOduration=1.28032422 podStartE2EDuration="1.28032422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:50.278021478 +0000 UTC m=+1610.606432875" watchObservedRunningTime="2026-10-30 19:03:50.28032422 +0000 UTC m=+1610.608735601" Oct 30 19:03:50.575419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.575337 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:03:50.583297 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.583267 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v"] Oct 30 19:03:50.588024 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.587996 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.604544 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.604517 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-e9ada0-openshift-default-dockercfg-dr8jc\"" Oct 30 19:03:50.626215 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.626186 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v"] Oct 30 19:03:50.690069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690038 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.690288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690132 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.690288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690163 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e661b30d-6632-43e8-9691-2d2006f50180-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.690288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690184 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.690478 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690353 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e661b30d-6632-43e8-9691-2d2006f50180-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.690478 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690397 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwk8h\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-kube-api-access-qwk8h\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.690658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690527 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.690658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690572 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.690795 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.690711 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.791995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.791956 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792178 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792087 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792178 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792144 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792271 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792233 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792370 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792276 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e661b30d-6632-43e8-9691-2d2006f50180-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792370 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792306 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792391 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e661b30d-6632-43e8-9691-2d2006f50180-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792412 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792425 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwk8h\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-kube-api-access-qwk8h\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792705 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792514 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792705 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792679 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792831 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792797 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.792932 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.792905 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.793206 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.793186 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e661b30d-6632-43e8-9691-2d2006f50180-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.794911 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.794881 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.795479 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.795460 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e661b30d-6632-43e8-9691-2d2006f50180-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.804976 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.804952 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.805297 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.805083 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwk8h\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-kube-api-access-qwk8h\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:50.904461 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:50.904418 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:51.204890 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:51.204849 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v"] Oct 30 19:03:51.205111 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:03:51.205087 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode661b30d_6632_43e8_9691_2d2006f50180.slice/crio-0acd40933e7af105c873aa7772a62a31859eb2e181e984bb29822546b1a63d3b WatchSource:0}: Error finding container 0acd40933e7af105c873aa7772a62a31859eb2e181e984bb29822546b1a63d3b: Status 404 returned error can't find the container with id 0acd40933e7af105c873aa7772a62a31859eb2e181e984bb29822546b1a63d3b Oct 30 19:03:51.208358 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:51.208317 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:03:51.208440 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:51.208415 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:03:51.208498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:51.208482 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:03:51.265275 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:51.265231 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" event={"ID":"e661b30d-6632-43e8-9691-2d2006f50180","Type":"ContainerStarted","Data":"0acd40933e7af105c873aa7772a62a31859eb2e181e984bb29822546b1a63d3b"} Oct 30 19:03:52.273984 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:52.273951 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" event={"ID":"e661b30d-6632-43e8-9691-2d2006f50180","Type":"ContainerStarted","Data":"c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66"} Oct 30 19:03:52.914633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:52.907001 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:52.914633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:52.912739 5959 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:03:52.914633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:52.912795 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:03:53.910631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:53.908024 5959 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:03:53.910631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:53.908086 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:03:54.937630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:54.937569 5959 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:03:54.938150 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:54.937648 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:03:55.917632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:55.907706 5959 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:03:55.917632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:55.907776 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:03:56.917094 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:56.917056 5959 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:03:56.917755 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:56.917719 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:03:57.781760 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:03:57.781725 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:03:57.911827 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:57.911796 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:57.912311 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:57.912286 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:57.913372 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:57.913349 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:03:57.937352 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:57.937299 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" podStartSLOduration=7.937284737 podStartE2EDuration="7.937284737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:52.319979852 +0000 UTC m=+1612.648391238" watchObservedRunningTime="2026-10-30 19:03:57.937284737 +0000 UTC m=+1618.265696118" Oct 30 19:03:58.920626 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:03:58.920266 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg"] Oct 30 19:04:09.375053 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:09.374595 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw"] Oct 30 19:04:11.078131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.078026 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m"] Oct 30 19:04:11.078941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.078909 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" podUID="708a6acc-01cf-43f2-9473-d71b91aa395a" containerName="maas-api" containerID="cri-o://24e72fedf1d30929bc5f70e9ba9a9248a29cf431be7a175b38c5bd48b8f617ce" gracePeriod=30 Oct 30 19:04:11.475790 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.475755 5959 generic.go:356] "Generic (PLEG): container finished" podID="708a6acc-01cf-43f2-9473-d71b91aa395a" containerID="24e72fedf1d30929bc5f70e9ba9a9248a29cf431be7a175b38c5bd48b8f617ce" exitCode=0 Oct 30 19:04:11.475971 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.475832 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" event={"ID":"708a6acc-01cf-43f2-9473-d71b91aa395a","Type":"ContainerDied","Data":"24e72fedf1d30929bc5f70e9ba9a9248a29cf431be7a175b38c5bd48b8f617ce"} Oct 30 19:04:11.539188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.539161 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:04:11.644626 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.644578 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls\") pod \"708a6acc-01cf-43f2-9473-d71b91aa395a\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " Oct 30 19:04:11.644841 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.644671 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708a6acc-01cf-43f2-9473-d71b91aa395a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708a6acc-01cf-43f2-9473-d71b91aa395a-openshift-service-ca\") pod \"708a6acc-01cf-43f2-9473-d71b91aa395a\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " Oct 30 19:04:11.644841 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.644692 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-metrics-certs\") pod \"708a6acc-01cf-43f2-9473-d71b91aa395a\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " Oct 30 19:04:11.644979 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.644853 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708a6acc-01cf-43f2-9473-d71b91aa395a-kube-api-access-pngpl\" (UniqueName: \"kubernetes.io/projected/708a6acc-01cf-43f2-9473-d71b91aa395a-kube-api-access-pngpl\") pod \"708a6acc-01cf-43f2-9473-d71b91aa395a\" (UID: \"708a6acc-01cf-43f2-9473-d71b91aa395a\") " Oct 30 19:04:11.645905 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.645871 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708a6acc-01cf-43f2-9473-d71b91aa395a-openshift-service-ca" pod "708a6acc-01cf-43f2-9473-d71b91aa395a" (UID: "708a6acc-01cf-43f2-9473-d71b91aa395a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:11.646898 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.646875 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708a6acc-01cf-43f2-9473-d71b91aa395a-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:11.647299 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.647275 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls" pod "708a6acc-01cf-43f2-9473-d71b91aa395a" (UID: "708a6acc-01cf-43f2-9473-d71b91aa395a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:11.647719 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.647689 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-metrics-certs" pod "708a6acc-01cf-43f2-9473-d71b91aa395a" (UID: "708a6acc-01cf-43f2-9473-d71b91aa395a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:11.648103 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.648075 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708a6acc-01cf-43f2-9473-d71b91aa395a-kube-api-access-pngpl" pod "708a6acc-01cf-43f2-9473-d71b91aa395a" (UID: "708a6acc-01cf-43f2-9473-d71b91aa395a"). InnerVolumeSpecName "kube-api-access-pngpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:11.748555 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.748518 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pngpl\" (UniqueName: \"kubernetes.io/projected/708a6acc-01cf-43f2-9473-d71b91aa395a-kube-api-access-pngpl\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:11.748555 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.748551 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:11.748555 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:11.748562 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708a6acc-01cf-43f2-9473-d71b91aa395a-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:12.483993 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:12.483966 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" Oct 30 19:04:12.483993 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:12.483983 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m" event={"ID":"708a6acc-01cf-43f2-9473-d71b91aa395a","Type":"ContainerDied","Data":"984f52895544a377106453202c10bcf769082a31aad95fd61531e8ae1f277f42"} Oct 30 19:04:12.484522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:12.484041 5959 scope.go:122] "RemoveContainer" containerID="24e72fedf1d30929bc5f70e9ba9a9248a29cf431be7a175b38c5bd48b8f617ce" Oct 30 19:04:12.502230 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:12.502198 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m"] Oct 30 19:04:12.506253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:12.506229 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e9ada0-7cbfc4cb8d-fmw7m"] Oct 30 19:04:13.798551 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:13.798516 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708a6acc-01cf-43f2-9473-d71b91aa395a" path="/var/lib/kubelet/pods/708a6acc-01cf-43f2-9473-d71b91aa395a/volumes" Oct 30 19:04:13.954138 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:13.954097 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" podUID="2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" containerName="main" containerID="cri-o://1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98" gracePeriod=45 Oct 30 19:04:14.289348 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.289323 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:04:14.375690 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.375565 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" podUID="67e9ccca-f161-44a9-a52e-627fc8fc5cf2" containerName="payload-pre-processing" containerID="cri-o://bca1721a36f6de4b7119202a56e54c7ec3d53d0ff52161b0d852c58cea5335c7" gracePeriod=25 Oct 30 19:04:14.391396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.391362 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-kube-api-access-xl8w8\" (UniqueName: \"kubernetes.io/projected/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-kube-api-access-xl8w8\") pod \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " Oct 30 19:04:14.391574 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.391470 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-dshm\") pod \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " Oct 30 19:04:14.391574 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.391562 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-model-cache\") pod \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " Oct 30 19:04:14.391711 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.391623 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tls-certs\") pod \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " Oct 30 19:04:14.391764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.391711 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tmp-dir\") pod \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " Oct 30 19:04:14.391874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.391850 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-home\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-home\") pod \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\" (UID: \"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b\") " Oct 30 19:04:14.391940 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.391917 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-model-cache" pod "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" (UID: "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:14.392207 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.392135 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-home" pod "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" (UID: "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:14.392207 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.392216 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tmp-dir" pod "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" (UID: "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:14.392542 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.392514 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:14.392542 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.392534 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:14.392748 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.392547 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:14.394074 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.394051 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-dshm" pod "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" (UID: "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:14.394577 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.394554 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tls-certs" pod "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" (UID: "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:14.394577 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.394559 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-kube-api-access-xl8w8" pod "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" (UID: "2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b"). InnerVolumeSpecName "kube-api-access-xl8w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:14.494202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.494166 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl8w8\" (UniqueName: \"kubernetes.io/projected/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-kube-api-access-xl8w8\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:14.494202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.494200 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:14.494202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.494210 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:14.500520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.500492 5959 generic.go:356] "Generic (PLEG): container finished" podID="67e9ccca-f161-44a9-a52e-627fc8fc5cf2" containerID="bca1721a36f6de4b7119202a56e54c7ec3d53d0ff52161b0d852c58cea5335c7" exitCode=0 Oct 30 19:04:14.500680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.500579 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" event={"ID":"67e9ccca-f161-44a9-a52e-627fc8fc5cf2","Type":"ContainerDied","Data":"bca1721a36f6de4b7119202a56e54c7ec3d53d0ff52161b0d852c58cea5335c7"} Oct 30 19:04:14.501992 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.501965 5959 generic.go:356] "Generic (PLEG): container finished" podID="2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" containerID="1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98" exitCode=0 Oct 30 19:04:14.502106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.502023 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" Oct 30 19:04:14.502106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.502046 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" event={"ID":"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b","Type":"ContainerDied","Data":"1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98"} Oct 30 19:04:14.502106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.502089 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg" event={"ID":"2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b","Type":"ContainerDied","Data":"ee95cfaa95054a92655f9f8f2cef095926cb1328da5d8fb77916716367abaaf5"} Oct 30 19:04:14.502219 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.502107 5959 scope.go:122] "RemoveContainer" containerID="1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98" Oct 30 19:04:14.516171 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.516151 5959 scope.go:122] "RemoveContainer" containerID="1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98" Oct 30 19:04:14.516556 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:04:14.516535 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98\": container with ID starting with 1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98 not found: ID does not exist" containerID="1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98" Oct 30 19:04:14.516675 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.516565 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98"} err="failed to get container status \"1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98\": rpc error: code = NotFound desc = could not find container \"1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98\": container with ID starting with 1970db0eac2a1bbe149dba8eb5cf80204fdb5059627c78acbb34a602a4c04a98 not found: ID does not exist" Oct 30 19:04:14.527304 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.527270 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg"] Oct 30 19:04:14.530598 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.530569 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-e9ada0/explicit-model-e9ada0-kserve-5dd59d7f75-kpftg"] Oct 30 19:04:14.639150 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.639123 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:04:14.696333 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.696293 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-plugins-config-volume\") pod \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\" (UID: \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\") " Oct 30 19:04:14.696622 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.696396 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-kube-api-access-g75fg\" (UniqueName: \"kubernetes.io/projected/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-kube-api-access-g75fg\") pod \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\" (UID: \"67e9ccca-f161-44a9-a52e-627fc8fc5cf2\") " Oct 30 19:04:14.696791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.696762 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-plugins-config-volume" pod "67e9ccca-f161-44a9-a52e-627fc8fc5cf2" (UID: "67e9ccca-f161-44a9-a52e-627fc8fc5cf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:14.697308 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.697280 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:14.698927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.698900 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-kube-api-access-g75fg" pod "67e9ccca-f161-44a9-a52e-627fc8fc5cf2" (UID: "67e9ccca-f161-44a9-a52e-627fc8fc5cf2"). InnerVolumeSpecName "kube-api-access-g75fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:14.800253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:14.800106 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75fg\" (UniqueName: \"kubernetes.io/projected/67e9ccca-f161-44a9-a52e-627fc8fc5cf2-kube-api-access-g75fg\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:15.512133 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:15.512102 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" Oct 30 19:04:15.512133 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:15.512115 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw" event={"ID":"67e9ccca-f161-44a9-a52e-627fc8fc5cf2","Type":"ContainerDied","Data":"447d015e867e89f03fe179278f5060857abdefe48574b610f4be88a98025ec3d"} Oct 30 19:04:15.512489 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:15.512178 5959 scope.go:122] "RemoveContainer" containerID="bca1721a36f6de4b7119202a56e54c7ec3d53d0ff52161b0d852c58cea5335c7" Oct 30 19:04:15.538025 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:15.537996 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw"] Oct 30 19:04:15.541978 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:15.541949 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e9ada0-558d6f86bc-wv7fw"] Oct 30 19:04:15.799647 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:15.799554 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" path="/var/lib/kubelet/pods/2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b/volumes" Oct 30 19:04:15.800040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:15.800024 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e9ccca-f161-44a9-a52e-627fc8fc5cf2" path="/var/lib/kubelet/pods/67e9ccca-f161-44a9-a52e-627fc8fc5cf2/volumes" Oct 30 19:04:16.979645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.979574 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg"] Oct 30 19:04:16.980259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980243 5959 cpu_manager.go:414] "containerMap: removing container" podUID="2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" containerName="main" Oct 30 19:04:16.980304 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980260 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" containerName="main" Oct 30 19:04:16.980345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980306 5959 cpu_manager.go:414] "containerMap: removing container" podUID="708a6acc-01cf-43f2-9473-d71b91aa395a" containerName="maas-api" Oct 30 19:04:16.980345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980312 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708a6acc-01cf-43f2-9473-d71b91aa395a" containerName="maas-api" Oct 30 19:04:16.980345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980325 5959 cpu_manager.go:414] "containerMap: removing container" podUID="67e9ccca-f161-44a9-a52e-627fc8fc5cf2" containerName="payload-pre-processing" Oct 30 19:04:16.980345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980330 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e9ccca-f161-44a9-a52e-627fc8fc5cf2" containerName="payload-pre-processing" Oct 30 19:04:16.980459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980434 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="708a6acc-01cf-43f2-9473-d71b91aa395a" containerName="maas-api" Oct 30 19:04:16.980459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980445 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e9ccca-f161-44a9-a52e-627fc8fc5cf2" containerName="payload-pre-processing" Oct 30 19:04:16.980459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.980455 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5f24dd-c99c-44aa-99a6-5a5cb0c2c51b" containerName="main" Oct 30 19:04:16.985525 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.985506 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:16.991462 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.991415 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c2e3f9-openshift-default-dockercfg-r9f7b\"" Oct 30 19:04:16.996038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:16.996012 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg"] Oct 30 19:04:17.037162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037127 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-token\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.037352 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037178 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-podinfo\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.037352 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037209 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lppd\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-kube-api-access-4lppd\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.037438 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037354 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-envoy\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.037438 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037411 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-credential-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.037576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037562 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-data\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.037634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037589 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.037674 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037656 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-certs\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.037736 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.037710 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.138815 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.138776 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.138879 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-token\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.138912 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-podinfo\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.138936 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lppd\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-kube-api-access-4lppd\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.138959 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-envoy\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.139288 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-credential-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.139459 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-data\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.139541 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.139567 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-credential-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139736 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.139594 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-certs\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139793 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.139764 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139837 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.139824 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-data\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.139966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.139946 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.140047 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.140027 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-certs\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.141906 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.141880 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-envoy\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.142074 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.142053 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-podinfo\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.147073 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.147049 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lppd\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-kube-api-access-4lppd\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.147157 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.147053 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-token\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.316164 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.316070 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:17.628072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.628020 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg"] Oct 30 19:04:17.635651 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.635579 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:04:17.635745 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.635679 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:04:17.635745 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:17.635711 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:04:18.550943 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:18.550903 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" event={"ID":"6bd33ce2-06d2-4680-af9a-58e00ea3ac03","Type":"ContainerStarted","Data":"b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b"} Oct 30 19:04:18.551445 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:18.550975 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" event={"ID":"6bd33ce2-06d2-4680-af9a-58e00ea3ac03","Type":"ContainerStarted","Data":"03bf028e254c7000fa4ee9c489d540b0cd885604f906590e493a3d79dca0c575"} Oct 30 19:04:18.578249 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:18.578185 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" podStartSLOduration=2.578168618 podStartE2EDuration="2.578168618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:18.57607245 +0000 UTC m=+1638.904483836" watchObservedRunningTime="2026-10-30 19:04:18.578168618 +0000 UTC m=+1638.906580008" Oct 30 19:04:19.141632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:19.141239 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v"] Oct 30 19:04:19.141632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:19.141589 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" containerID="cri-o://c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66" gracePeriod=30 Oct 30 19:04:19.320629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:19.319941 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:19.320629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:19.320105 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 19:04:19.320629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:19.320182 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 19:04:20.250674 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.250256 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc"] Oct 30 19:04:20.256373 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.255954 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.270664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.268657 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc"] Oct 30 19:04:20.317101 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.317056 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 19:04:20.326695 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.322466 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 19:04:20.326940 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.322377 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-754cee-openshift-default-dockercfg-lb5bl\"" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.332753 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-socket\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.335681 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e811685a-9b1e-40af-9d49-000e14b4b158-istiod-ca-cert\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.335727 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-data\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.335861 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e811685a-9b1e-40af-9d49-000e14b4b158-istio-podinfo\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.336001 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2fd\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-kube-api-access-kr2fd\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.336046 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-istio-token\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.336094 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-envoy\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.336140 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-certs\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.345671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.336231 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-credential-socket\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.438625 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2fd\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-kube-api-access-kr2fd\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.461376 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-istio-token\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.461540 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-envoy\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.461677 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-certs\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.461966 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-credential-socket\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.462267 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-socket\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.462311 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e811685a-9b1e-40af-9d49-000e14b4b158-istiod-ca-cert\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.462439 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-data\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.462885 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e811685a-9b1e-40af-9d49-000e14b4b158-istio-podinfo\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.467018 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-certs\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.467774 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-socket\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.467999 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-credential-socket\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.468585 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e811685a-9b1e-40af-9d49-000e14b4b158-istiod-ca-cert\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.473629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.468844 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-data\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.488128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.483953 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-envoy\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.488128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.484674 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e811685a-9b1e-40af-9d49-000e14b4b158-istio-podinfo\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.488128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.485332 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2fd\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-kube-api-access-kr2fd\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.488128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.487869 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-istio-token\") pod \"e2e-noten-754cee-openshift-default-54666dcff9-k8xdc\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.576028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.575935 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:20.929965 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.929933 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc"] Oct 30 19:04:20.931571 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:04:20.931187 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode811685a_9b1e_40af_9d49_000e14b4b158.slice/crio-6c22bf1d4b1bca7e5932c2ed03970b9e9cb79010768ffc3d39afb7e878a5aa14 WatchSource:0}: Error finding container 6c22bf1d4b1bca7e5932c2ed03970b9e9cb79010768ffc3d39afb7e878a5aa14: Status 404 returned error can't find the container with id 6c22bf1d4b1bca7e5932c2ed03970b9e9cb79010768ffc3d39afb7e878a5aa14 Oct 30 19:04:20.934362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.933930 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:04:20.934362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.933997 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:04:20.934362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:20.934029 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:04:21.320758 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:21.320655 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 19:04:21.321386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:21.321340 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 19:04:21.580963 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:21.580862 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" event={"ID":"e811685a-9b1e-40af-9d49-000e14b4b158","Type":"ContainerStarted","Data":"2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f"} Oct 30 19:04:21.581120 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:21.580988 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" event={"ID":"e811685a-9b1e-40af-9d49-000e14b4b158","Type":"ContainerStarted","Data":"6c22bf1d4b1bca7e5932c2ed03970b9e9cb79010768ffc3d39afb7e878a5aa14"} Oct 30 19:04:21.625763 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:21.610111 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" podStartSLOduration=1.610089507 podStartE2EDuration="1.610089507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:21.60902687 +0000 UTC m=+1641.937438257" watchObservedRunningTime="2026-10-30 19:04:21.610089507 +0000 UTC m=+1641.938500894" Oct 30 19:04:22.318630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:22.317844 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 19:04:22.318630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:22.317925 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 19:04:22.577554 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:22.577218 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:22.586480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:22.586449 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:22.593482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:22.593455 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:22.602991 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:22.602957 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:23.319050 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.319013 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 19:04:23.319387 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.319358 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 19:04:23.766984 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.766953 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb"] Oct 30 19:04:23.771742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.771723 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.798772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.798586 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-754cee\"/\"noten-model-754cee-kserve-self-signed-certs\"" Oct 30 19:04:23.798772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.798688 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-754cee\"/\"kube-root-ca.crt\"" Oct 30 19:04:23.799033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.798785 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-754cee\"/\"default-dockercfg-q9g7n\"" Oct 30 19:04:23.803400 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.803370 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-754cee\"/\"openshift-service-ca.crt\"" Oct 30 19:04:23.808362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.808336 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb"] Oct 30 19:04:23.854768 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.854625 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srmpd\" (UniqueName: \"kubernetes.io/projected/cc2fca9d-5bdb-4887-9567-4389ec1e8278-kube-api-access-srmpd\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.854768 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.854744 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tls-certs\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.855016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.854830 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-dshm\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.855065 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.855028 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-model-cache\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.855065 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.855062 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-home\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.855142 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.855104 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tmp-dir\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.956657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.956594 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-model-cache\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.956833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.956682 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-home\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.956833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.956728 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tmp-dir\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.956833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.956798 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srmpd\" (UniqueName: \"kubernetes.io/projected/cc2fca9d-5bdb-4887-9567-4389ec1e8278-kube-api-access-srmpd\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.957013 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.956846 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tls-certs\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.957013 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.956929 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-dshm\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.957119 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.957086 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-model-cache\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.957573 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.957546 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tmp-dir\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.957758 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.957575 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-home\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.960121 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.960091 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-dshm\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.960494 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.960473 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tls-certs\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:23.973706 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:23.973675 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srmpd\" (UniqueName: \"kubernetes.io/projected/cc2fca9d-5bdb-4887-9567-4389ec1e8278-kube-api-access-srmpd\") pod \"noten-model-754cee-kserve-f94b66f49-2pmgb\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:24.100092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.096964 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:24.444209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.444177 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:24.445665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.445628 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:24.448913 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.448889 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:04:24.493569 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.493541 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:04:24.507538 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.507507 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb"] Oct 30 19:04:24.571530 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.571497 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-istio-token\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-istio-token\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.571916 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.571891 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-socket\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.572154 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572136 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-certs\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572255 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-socket" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572271 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-credential-socket\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572334 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e661b30d-6632-43e8-9691-2d2006f50180-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e661b30d-6632-43e8-9691-2d2006f50180-istiod-ca-cert\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572364 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-envoy\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572390 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-data\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572428 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e661b30d-6632-43e8-9691-2d2006f50180-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e661b30d-6632-43e8-9691-2d2006f50180-istio-podinfo\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572661 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-kube-api-access-qwk8h\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-kube-api-access-qwk8h\") pod \"e661b30d-6632-43e8-9691-2d2006f50180\" (UID: \"e661b30d-6632-43e8-9691-2d2006f50180\") " Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572705 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e661b30d-6632-43e8-9691-2d2006f50180-istiod-ca-cert" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.572945 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-credential-socket" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.573491 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.573512 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e661b30d-6632-43e8-9691-2d2006f50180-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.573528 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:24.574019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.573980 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-certs" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:24.575047 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.574030 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-istio-token" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:24.575208 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.575161 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-data" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:24.575208 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.575181 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-kube-api-access-qwk8h" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "kube-api-access-qwk8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:24.584496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.584446 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e661b30d-6632-43e8-9691-2d2006f50180-istio-podinfo" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:04:24.585447 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.585411 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-envoy" pod "e661b30d-6632-43e8-9691-2d2006f50180" (UID: "e661b30d-6632-43e8-9691-2d2006f50180"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:24.621849 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.621808 5959 generic.go:356] "Generic (PLEG): container finished" podID="e661b30d-6632-43e8-9691-2d2006f50180" containerID="c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66" exitCode=0 Oct 30 19:04:24.622033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.621853 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" Oct 30 19:04:24.622033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.621904 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" event={"ID":"e661b30d-6632-43e8-9691-2d2006f50180","Type":"ContainerDied","Data":"c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66"} Oct 30 19:04:24.622033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.621989 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v" event={"ID":"e661b30d-6632-43e8-9691-2d2006f50180","Type":"ContainerDied","Data":"0acd40933e7af105c873aa7772a62a31859eb2e181e984bb29822546b1a63d3b"} Oct 30 19:04:24.622033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.622022 5959 scope.go:122] "RemoveContainer" containerID="c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66" Oct 30 19:04:24.626435 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.626403 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" event={"ID":"cc2fca9d-5bdb-4887-9567-4389ec1e8278","Type":"ContainerStarted","Data":"66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859"} Oct 30 19:04:24.626577 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.626477 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" event={"ID":"cc2fca9d-5bdb-4887-9567-4389ec1e8278","Type":"ContainerStarted","Data":"a46e679dd4eeaa75252be97d662938d90a3e910381129ad8a4a025d1b4839b56"} Oct 30 19:04:24.637576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.637558 5959 scope.go:122] "RemoveContainer" containerID="c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66" Oct 30 19:04:24.637914 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:04:24.637895 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66\": container with ID starting with c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66 not found: ID does not exist" containerID="c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66" Oct 30 19:04:24.637972 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.637923 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66"} err="failed to get container status \"c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66\": rpc error: code = NotFound desc = could not find container \"c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66\": container with ID starting with c8b5558d9add4dc6835e73e27dc72457c9b5dfbe0c808f5577f51902a2f3ff66 not found: ID does not exist" Oct 30 19:04:24.647176 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.647127 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" podStartSLOduration=1.6471140659999999 podStartE2EDuration="1.647114066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:24.642375373 +0000 UTC m=+1644.970786760" watchObservedRunningTime="2026-10-30 19:04:24.647114066 +0000 UTC m=+1644.975525450" Oct 30 19:04:24.661008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.660979 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v"] Oct 30 19:04:24.665086 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.665062 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-vrq7v"] Oct 30 19:04:24.675955 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.675922 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:24.676175 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.676151 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:24.676175 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.676177 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e661b30d-6632-43e8-9691-2d2006f50180-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:24.676325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.676192 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e661b30d-6632-43e8-9691-2d2006f50180-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:24.676325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.676209 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwk8h\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-kube-api-access-qwk8h\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:24.676325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:24.676222 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e661b30d-6632-43e8-9691-2d2006f50180-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:25.097612 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:25.097583 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:25.798277 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:25.798242 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e661b30d-6632-43e8-9691-2d2006f50180" path="/var/lib/kubelet/pods/e661b30d-6632-43e8-9691-2d2006f50180/volumes" Oct 30 19:04:27.782028 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:04:27.781978 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:04:34.013439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:34.013389 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb"] Oct 30 19:04:34.171876 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:34.171823 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc"] Oct 30 19:04:34.172724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:34.172682 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" podUID="e811685a-9b1e-40af-9d49-000e14b4b158" containerName="istio-proxy" containerID="cri-o://2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f" gracePeriod=30 Oct 30 19:04:37.596271 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:37.596227 5959 patch_prober.go:29] interesting pod/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 19:04:37.596797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:37.596301 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" podUID="e811685a-9b1e-40af-9d49-000e14b4b158" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:04:39.484398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.484325 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:39.560020 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.559988 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-socket\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.560201 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560052 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-credential-socket\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.560201 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560091 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e811685a-9b1e-40af-9d49-000e14b4b158-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e811685a-9b1e-40af-9d49-000e14b4b158-istio-podinfo\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.560201 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560123 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-envoy\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.560201 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560185 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-istio-token\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-istio-token\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.560424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560308 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e811685a-9b1e-40af-9d49-000e14b4b158-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e811685a-9b1e-40af-9d49-000e14b4b158-istiod-ca-cert\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.560424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560339 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-socket" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:39.560424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560411 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-certs\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.560573 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560444 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-credential-socket" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:39.560573 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560498 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-kube-api-access-kr2fd\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-kube-api-access-kr2fd\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.560573 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.560550 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-data\") pod \"e811685a-9b1e-40af-9d49-000e14b4b158\" (UID: \"e811685a-9b1e-40af-9d49-000e14b4b158\") " Oct 30 19:04:39.561317 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.561078 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-certs" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:39.561317 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.561242 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.561317 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.561262 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.561317 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.561276 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.561839 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.561478 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-data" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:39.562079 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.562050 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e811685a-9b1e-40af-9d49-000e14b4b158-istiod-ca-cert" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:39.563334 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.563309 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-envoy" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:39.563448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.563432 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-istio-token" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:39.563982 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.563958 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-kube-api-access-kr2fd" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "kube-api-access-kr2fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:39.564279 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.564250 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e811685a-9b1e-40af-9d49-000e14b4b158-istio-podinfo" pod "e811685a-9b1e-40af-9d49-000e14b4b158" (UID: "e811685a-9b1e-40af-9d49-000e14b4b158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:04:39.662450 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.662413 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.662450 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.662446 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e811685a-9b1e-40af-9d49-000e14b4b158-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.662450 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.662457 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e811685a-9b1e-40af-9d49-000e14b4b158-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.662719 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.662466 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.662719 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.662476 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e811685a-9b1e-40af-9d49-000e14b4b158-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.662719 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.662486 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr2fd\" (UniqueName: \"kubernetes.io/projected/e811685a-9b1e-40af-9d49-000e14b4b158-kube-api-access-kr2fd\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.756073 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.748512 5959 generic.go:356] "Generic (PLEG): container finished" podID="e811685a-9b1e-40af-9d49-000e14b4b158" containerID="2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f" exitCode=0 Oct 30 19:04:39.756073 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.748581 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" event={"ID":"e811685a-9b1e-40af-9d49-000e14b4b158","Type":"ContainerDied","Data":"2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f"} Oct 30 19:04:39.756073 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.748650 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" event={"ID":"e811685a-9b1e-40af-9d49-000e14b4b158","Type":"ContainerDied","Data":"6c22bf1d4b1bca7e5932c2ed03970b9e9cb79010768ffc3d39afb7e878a5aa14"} Oct 30 19:04:39.756073 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.748679 5959 scope.go:122] "RemoveContainer" containerID="2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f" Oct 30 19:04:39.756073 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.748862 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc" Oct 30 19:04:39.768467 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.768432 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:04:39.785040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.785012 5959 scope.go:122] "RemoveContainer" containerID="2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f" Oct 30 19:04:39.785454 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:04:39.785427 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f\": container with ID starting with 2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f not found: ID does not exist" containerID="2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f" Oct 30 19:04:39.785566 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.785460 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f"} err="failed to get container status \"2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f\": rpc error: code = NotFound desc = could not find container \"2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f\": container with ID starting with 2c9df7dd4e130b3fb1f5dae924b1c94d63f55fb58e6817b7510cc97f65f64e6f not found: ID does not exist" Oct 30 19:04:39.802009 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.801965 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc"] Oct 30 19:04:39.807242 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:39.807203 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-754cee-openshift-default-54666dcff9-k8xdc"] Oct 30 19:04:41.800972 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:41.800937 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e811685a-9b1e-40af-9d49-000e14b4b158" path="/var/lib/kubelet/pods/e811685a-9b1e-40af-9d49-000e14b4b158/volumes" Oct 30 19:04:49.054134 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.051991 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" podUID="cc2fca9d-5bdb-4887-9567-4389ec1e8278" containerName="main" containerID="cri-o://66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859" gracePeriod=45 Oct 30 19:04:49.341639 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.341594 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:49.507768 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.507735 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2fca9d-5bdb-4887-9567-4389ec1e8278-kube-api-access-srmpd\" (UniqueName: \"kubernetes.io/projected/cc2fca9d-5bdb-4887-9567-4389ec1e8278-kube-api-access-srmpd\") pod \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " Oct 30 19:04:49.507946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.507849 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-model-cache\") pod \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " Oct 30 19:04:49.508015 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.507997 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-home\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-home\") pod \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " Oct 30 19:04:49.508056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508034 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-dshm\") pod \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " Oct 30 19:04:49.508093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508072 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tls-certs\") pod \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " Oct 30 19:04:49.508133 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508079 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-model-cache" pod "cc2fca9d-5bdb-4887-9567-4389ec1e8278" (UID: "cc2fca9d-5bdb-4887-9567-4389ec1e8278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:49.508188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508175 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tmp-dir\") pod \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\" (UID: \"cc2fca9d-5bdb-4887-9567-4389ec1e8278\") " Oct 30 19:04:49.508290 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508265 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-home" pod "cc2fca9d-5bdb-4887-9567-4389ec1e8278" (UID: "cc2fca9d-5bdb-4887-9567-4389ec1e8278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:49.508429 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508408 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tmp-dir" pod "cc2fca9d-5bdb-4887-9567-4389ec1e8278" (UID: "cc2fca9d-5bdb-4887-9567-4389ec1e8278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:49.508878 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508860 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:49.508981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508896 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:49.508981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.508905 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:49.510330 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.510303 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-dshm" pod "cc2fca9d-5bdb-4887-9567-4389ec1e8278" (UID: "cc2fca9d-5bdb-4887-9567-4389ec1e8278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:49.510426 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.510372 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tls-certs" pod "cc2fca9d-5bdb-4887-9567-4389ec1e8278" (UID: "cc2fca9d-5bdb-4887-9567-4389ec1e8278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:49.510426 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.510409 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2fca9d-5bdb-4887-9567-4389ec1e8278-kube-api-access-srmpd" pod "cc2fca9d-5bdb-4887-9567-4389ec1e8278" (UID: "cc2fca9d-5bdb-4887-9567-4389ec1e8278"). InnerVolumeSpecName "kube-api-access-srmpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:49.610345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.610261 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2fca9d-5bdb-4887-9567-4389ec1e8278-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:49.610345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.610294 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2fca9d-5bdb-4887-9567-4389ec1e8278-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:49.610345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.610305 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srmpd\" (UniqueName: \"kubernetes.io/projected/cc2fca9d-5bdb-4887-9567-4389ec1e8278-kube-api-access-srmpd\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:49.827137 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.827103 5959 generic.go:356] "Generic (PLEG): container finished" podID="cc2fca9d-5bdb-4887-9567-4389ec1e8278" containerID="66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859" exitCode=0 Oct 30 19:04:49.827335 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.827146 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" Oct 30 19:04:49.827335 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.827168 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" event={"ID":"cc2fca9d-5bdb-4887-9567-4389ec1e8278","Type":"ContainerDied","Data":"66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859"} Oct 30 19:04:49.827335 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.827206 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb" event={"ID":"cc2fca9d-5bdb-4887-9567-4389ec1e8278","Type":"ContainerDied","Data":"a46e679dd4eeaa75252be97d662938d90a3e910381129ad8a4a025d1b4839b56"} Oct 30 19:04:49.827335 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.827234 5959 scope.go:122] "RemoveContainer" containerID="66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859" Oct 30 19:04:49.837371 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.837349 5959 scope.go:122] "RemoveContainer" containerID="66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859" Oct 30 19:04:49.837714 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:04:49.837688 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859\": container with ID starting with 66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859 not found: ID does not exist" containerID="66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859" Oct 30 19:04:49.837796 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.837728 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859"} err="failed to get container status \"66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859\": rpc error: code = NotFound desc = could not find container \"66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859\": container with ID starting with 66022288bb9a034e2ea1185dc0e3706a79bf9fd3dbae8fb77b0263eba45f2859 not found: ID does not exist" Oct 30 19:04:49.870806 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.870711 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb"] Oct 30 19:04:49.883567 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:49.883535 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-754cee/noten-model-754cee-kserve-f94b66f49-2pmgb"] Oct 30 19:04:51.804981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:04:51.804935 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2fca9d-5bdb-4887-9567-4389ec1e8278" path="/var/lib/kubelet/pods/cc2fca9d-5bdb-4887-9567-4389ec1e8278/volumes" Oct 30 19:04:57.781092 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:04:57.781055 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:05:27.447950 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:27.447918 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:05:27.782592 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:05:27.782508 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:05:29.800888 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.800857 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj"] Oct 30 19:05:29.801449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801435 5959 cpu_manager.go:414] "containerMap: removing container" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" Oct 30 19:05:29.801498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801450 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" Oct 30 19:05:29.801498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801465 5959 cpu_manager.go:414] "containerMap: removing container" podUID="cc2fca9d-5bdb-4887-9567-4389ec1e8278" containerName="main" Oct 30 19:05:29.801498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801470 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2fca9d-5bdb-4887-9567-4389ec1e8278" containerName="main" Oct 30 19:05:29.801498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801484 5959 cpu_manager.go:414] "containerMap: removing container" podUID="e811685a-9b1e-40af-9d49-000e14b4b158" containerName="istio-proxy" Oct 30 19:05:29.801498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801488 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e811685a-9b1e-40af-9d49-000e14b4b158" containerName="istio-proxy" Oct 30 19:05:29.801713 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801620 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e661b30d-6632-43e8-9691-2d2006f50180" containerName="istio-proxy" Oct 30 19:05:29.801713 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801632 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2fca9d-5bdb-4887-9567-4389ec1e8278" containerName="main" Oct 30 19:05:29.801713 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.801645 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e811685a-9b1e-40af-9d49-000e14b4b158" containerName="istio-proxy" Oct 30 19:05:29.805674 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.805652 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:29.817477 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.817444 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj"] Oct 30 19:05:29.817790 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.817764 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-dcd326\"" Oct 30 19:05:29.818132 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.818114 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-dcd326-dockercfg-dbr8p\"" Oct 30 19:05:29.860827 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.860786 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz"] Oct 30 19:05:29.865582 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.865557 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:29.901422 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.901386 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz"] Oct 30 19:05:29.954587 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.954534 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4db960a-a125-48a3-a095-0cbc7d6b79b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz\" (UID: \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:29.955159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.955137 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthtn\" (UniqueName: \"kubernetes.io/projected/e4db960a-a125-48a3-a095-0cbc7d6b79b3-kube-api-access-vthtn\") pod \"payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz\" (UID: \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:29.955569 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.955549 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb38225-6af5-4a9a-824a-2e232854b71a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj\" (UID: \"ddb38225-6af5-4a9a-824a-2e232854b71a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:29.956087 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:29.956066 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9sx\" (UniqueName: \"kubernetes.io/projected/ddb38225-6af5-4a9a-824a-2e232854b71a-kube-api-access-cj9sx\") pod \"payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj\" (UID: \"ddb38225-6af5-4a9a-824a-2e232854b71a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:30.057650 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.057531 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthtn\" (UniqueName: \"kubernetes.io/projected/e4db960a-a125-48a3-a095-0cbc7d6b79b3-kube-api-access-vthtn\") pod \"payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz\" (UID: \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:30.057837 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.057780 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb38225-6af5-4a9a-824a-2e232854b71a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj\" (UID: \"ddb38225-6af5-4a9a-824a-2e232854b71a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:30.057942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.057927 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9sx\" (UniqueName: \"kubernetes.io/projected/ddb38225-6af5-4a9a-824a-2e232854b71a-kube-api-access-cj9sx\") pod \"payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj\" (UID: \"ddb38225-6af5-4a9a-824a-2e232854b71a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:30.066635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.059103 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4db960a-a125-48a3-a095-0cbc7d6b79b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz\" (UID: \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:30.066881 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.066714 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4db960a-a125-48a3-a095-0cbc7d6b79b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz\" (UID: \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:30.067340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.067302 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb38225-6af5-4a9a-824a-2e232854b71a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj\" (UID: \"ddb38225-6af5-4a9a-824a-2e232854b71a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:30.075624 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.071588 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthtn\" (UniqueName: \"kubernetes.io/projected/e4db960a-a125-48a3-a095-0cbc7d6b79b3-kube-api-access-vthtn\") pod \"payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz\" (UID: \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:30.077411 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.077385 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9sx\" (UniqueName: \"kubernetes.io/projected/ddb38225-6af5-4a9a-824a-2e232854b71a-kube-api-access-cj9sx\") pod \"payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj\" (UID: \"ddb38225-6af5-4a9a-824a-2e232854b71a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:30.121696 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.121662 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:30.208324 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.208283 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:30.425306 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.425272 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj"] Oct 30 19:05:30.426083 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:05:30.426053 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb38225_6af5_4a9a_824a_2e232854b71a.slice/crio-635176423f578167d27df554c0e5ab628536c7f91364ef64719278fe7c3545d6 WatchSource:0}: Error finding container 635176423f578167d27df554c0e5ab628536c7f91364ef64719278fe7c3545d6: Status 404 returned error can't find the container with id 635176423f578167d27df554c0e5ab628536c7f91364ef64719278fe7c3545d6 Oct 30 19:05:30.516590 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:30.516565 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz"] Oct 30 19:05:30.518065 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:05:30.518039 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4db960a_a125_48a3_a095_0cbc7d6b79b3.slice/crio-ad4ea73eb3ada52c94d0a40fb6fddf7b6b0312eb4befa3a10e98212bd326d267 WatchSource:0}: Error finding container ad4ea73eb3ada52c94d0a40fb6fddf7b6b0312eb4befa3a10e98212bd326d267: Status 404 returned error can't find the container with id ad4ea73eb3ada52c94d0a40fb6fddf7b6b0312eb4befa3a10e98212bd326d267 Oct 30 19:05:31.157237 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:31.157197 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" event={"ID":"e4db960a-a125-48a3-a095-0cbc7d6b79b3","Type":"ContainerStarted","Data":"f434c249255ee29c907e2b0935024903ab8e68667355e80f6154cbfb2aa0fd09"} Oct 30 19:05:31.157787 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:31.157267 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:31.157787 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:31.157290 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" event={"ID":"e4db960a-a125-48a3-a095-0cbc7d6b79b3","Type":"ContainerStarted","Data":"ad4ea73eb3ada52c94d0a40fb6fddf7b6b0312eb4befa3a10e98212bd326d267"} Oct 30 19:05:31.158968 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:31.158945 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" event={"ID":"ddb38225-6af5-4a9a-824a-2e232854b71a","Type":"ContainerStarted","Data":"f4ff0a5f28706481ce45aba8dd9b49703dcbd9f247664d3294aedb24118315e3"} Oct 30 19:05:31.159098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:31.158983 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" event={"ID":"ddb38225-6af5-4a9a-824a-2e232854b71a","Type":"ContainerStarted","Data":"635176423f578167d27df554c0e5ab628536c7f91364ef64719278fe7c3545d6"} Oct 30 19:05:31.159098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:31.159035 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:31.199113 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:31.199056 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" podStartSLOduration=2.199038338 podStartE2EDuration="2.199038338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:31.198793127 +0000 UTC m=+1711.527204513" watchObservedRunningTime="2026-10-30 19:05:31.199038338 +0000 UTC m=+1711.527449724" Oct 30 19:05:31.245771 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:31.245717 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" podStartSLOduration=2.245700513 podStartE2EDuration="2.245700513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:31.243669061 +0000 UTC m=+1711.572080440" watchObservedRunningTime="2026-10-30 19:05:31.245700513 +0000 UTC m=+1711.574111898" Oct 30 19:05:38.828688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:38.828652 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh"] Oct 30 19:05:38.833594 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:38.833565 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:38.841249 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:38.840999 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-dcd326-serving-cert\"" Oct 30 19:05:38.847337 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:38.847307 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh"] Oct 30 19:05:38.918081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:38.918049 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb849\" (UniqueName: \"kubernetes.io/projected/e9633de6-8e2a-45ee-a960-5c6e797c384b-kube-api-access-bb849\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:38.918262 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:38.918105 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:38.918348 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:38.918323 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:38.918434 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:38.918415 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9633de6-8e2a-45ee-a960-5c6e797c384b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.024667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.019715 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.024667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.019803 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9633de6-8e2a-45ee-a960-5c6e797c384b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.024667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.019955 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb849\" (UniqueName: \"kubernetes.io/projected/e9633de6-8e2a-45ee-a960-5c6e797c384b-kube-api-access-bb849\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.024667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.020011 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.024667 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:05:39.020198 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-serving-cert: secret "maas-api-e2e-shared-a-w1-dcd326-serving-cert" not found Oct 30 19:05:39.024667 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:05:39.020266 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls podName:e9633de6-8e2a-45ee-a960-5c6e797c384b nodeName:}" failed. No retries permitted until 2026-10-30 19:05:39.520243664 +0000 UTC m=+1719.848655040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls") pod "maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" (UID: "e9633de6-8e2a-45ee-a960-5c6e797c384b") : secret "maas-api-e2e-shared-a-w1-dcd326-serving-cert" not found Oct 30 19:05:39.024667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.020832 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9633de6-8e2a-45ee-a960-5c6e797c384b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.024667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.023756 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.033009 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.032975 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb849\" (UniqueName: \"kubernetes.io/projected/e9633de6-8e2a-45ee-a960-5c6e797c384b-kube-api-access-bb849\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.535095 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.535040 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.539281 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.539241 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:39.758937 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:39.758901 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:40.066088 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:40.066036 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh"] Oct 30 19:05:40.068650 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:05:40.068616 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9633de6_8e2a_45ee_a960_5c6e797c384b.slice/crio-b1eaa7c694f90ed7eb1143480a12bc3d999ffef80b4f895540f2893aed4eaca0 WatchSource:0}: Error finding container b1eaa7c694f90ed7eb1143480a12bc3d999ffef80b4f895540f2893aed4eaca0: Status 404 returned error can't find the container with id b1eaa7c694f90ed7eb1143480a12bc3d999ffef80b4f895540f2893aed4eaca0 Oct 30 19:05:40.240405 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:40.240371 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" event={"ID":"e9633de6-8e2a-45ee-a960-5c6e797c384b","Type":"ContainerStarted","Data":"b1eaa7c694f90ed7eb1143480a12bc3d999ffef80b4f895540f2893aed4eaca0"} Oct 30 19:05:41.249125 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:41.249088 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" event={"ID":"e9633de6-8e2a-45ee-a960-5c6e797c384b","Type":"ContainerStarted","Data":"18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2"} Oct 30 19:05:41.249595 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:41.249155 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:41.272314 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:41.272245 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" podStartSLOduration=2.78982379 podStartE2EDuration="3.272226621s" totalImagesPullingTime="482.402831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:41.267767728 +0000 UTC m=+1721.596179113" watchObservedRunningTime="2026-10-30 19:05:41.272226621 +0000 UTC m=+1721.600638008" Oct 30 19:05:42.178249 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:42.178216 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:05:42.178440 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:42.178415 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:05:47.266798 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:47.266762 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:05:57.783553 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:05:57.783510 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:05:59.461530 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.461476 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw"] Oct 30 19:05:59.466657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.466626 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:05:59.475797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.475765 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-46385c\"" Oct 30 19:05:59.476596 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.476306 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-46385c-dockercfg-ftmc7\"" Oct 30 19:05:59.482287 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.482254 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx"] Oct 30 19:05:59.490776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.488647 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw"] Oct 30 19:05:59.490776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.488796 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:05:59.501054 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.501025 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx"] Oct 30 19:05:59.563829 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.563777 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgj2b\" (UniqueName: \"kubernetes.io/projected/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-kube-api-access-hgj2b\") pod \"payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx\" (UID: \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:05:59.564022 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.563999 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ce5351-e16e-498b-a06d-8a70ca71768b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw\" (UID: \"49ce5351-e16e-498b-a06d-8a70ca71768b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:05:59.564258 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.564232 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfqvv\" (UniqueName: \"kubernetes.io/projected/49ce5351-e16e-498b-a06d-8a70ca71768b-kube-api-access-rfqvv\") pod \"payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw\" (UID: \"49ce5351-e16e-498b-a06d-8a70ca71768b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:05:59.564359 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.564303 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx\" (UID: \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:05:59.675912 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.675869 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgj2b\" (UniqueName: \"kubernetes.io/projected/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-kube-api-access-hgj2b\") pod \"payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx\" (UID: \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:05:59.676624 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.676577 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ce5351-e16e-498b-a06d-8a70ca71768b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw\" (UID: \"49ce5351-e16e-498b-a06d-8a70ca71768b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:05:59.676954 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.676935 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqvv\" (UniqueName: \"kubernetes.io/projected/49ce5351-e16e-498b-a06d-8a70ca71768b-kube-api-access-rfqvv\") pod \"payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw\" (UID: \"49ce5351-e16e-498b-a06d-8a70ca71768b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:05:59.677145 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.677125 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx\" (UID: \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:05:59.678292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.678269 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx\" (UID: \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:05:59.699811 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.699774 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ce5351-e16e-498b-a06d-8a70ca71768b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw\" (UID: \"49ce5351-e16e-498b-a06d-8a70ca71768b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:05:59.707155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.703849 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgj2b\" (UniqueName: \"kubernetes.io/projected/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-kube-api-access-hgj2b\") pod \"payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx\" (UID: \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:05:59.731079 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.727963 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfqvv\" (UniqueName: \"kubernetes.io/projected/49ce5351-e16e-498b-a06d-8a70ca71768b-kube-api-access-rfqvv\") pod \"payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw\" (UID: \"49ce5351-e16e-498b-a06d-8a70ca71768b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:05:59.800682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.799952 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:05:59.834986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:05:59.834677 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:06:00.140042 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:06:00.140009 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ce5351_e16e_498b_a06d_8a70ca71768b.slice/crio-8ad0cfba0c44a751e1fa703b3b1d94a580d9c3ce07ae2a00c9b2a54726da54c7 WatchSource:0}: Error finding container 8ad0cfba0c44a751e1fa703b3b1d94a580d9c3ce07ae2a00c9b2a54726da54c7: Status 404 returned error can't find the container with id 8ad0cfba0c44a751e1fa703b3b1d94a580d9c3ce07ae2a00c9b2a54726da54c7 Oct 30 19:06:00.140450 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.140429 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw"] Oct 30 19:06:00.155393 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.155362 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx"] Oct 30 19:06:00.157372 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:06:00.157345 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb94bfc_8707_4375_9f7f_5c454fcb06e2.slice/crio-dd9fd2f27bc4a0ae0624be7af58ad01a04a92f10be8e2c22ab9cbfea59668594 WatchSource:0}: Error finding container dd9fd2f27bc4a0ae0624be7af58ad01a04a92f10be8e2c22ab9cbfea59668594: Status 404 returned error can't find the container with id dd9fd2f27bc4a0ae0624be7af58ad01a04a92f10be8e2c22ab9cbfea59668594 Oct 30 19:06:00.397856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.397751 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" event={"ID":"5eb94bfc-8707-4375-9f7f-5c454fcb06e2","Type":"ContainerStarted","Data":"f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d"} Oct 30 19:06:00.397856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.397818 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" event={"ID":"5eb94bfc-8707-4375-9f7f-5c454fcb06e2","Type":"ContainerStarted","Data":"dd9fd2f27bc4a0ae0624be7af58ad01a04a92f10be8e2c22ab9cbfea59668594"} Oct 30 19:06:00.398092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.397963 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:06:00.399706 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.399676 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" event={"ID":"49ce5351-e16e-498b-a06d-8a70ca71768b","Type":"ContainerStarted","Data":"8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8"} Oct 30 19:06:00.399870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.399722 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" event={"ID":"49ce5351-e16e-498b-a06d-8a70ca71768b","Type":"ContainerStarted","Data":"8ad0cfba0c44a751e1fa703b3b1d94a580d9c3ce07ae2a00c9b2a54726da54c7"} Oct 30 19:06:00.399870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.399778 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:06:00.415279 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:00.415207 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" podStartSLOduration=1.415188966 podStartE2EDuration="1.415188966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:00.41332529 +0000 UTC m=+1740.741736677" watchObservedRunningTime="2026-10-30 19:06:00.415188966 +0000 UTC m=+1740.743600351" Oct 30 19:06:09.916207 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:09.916137 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" podStartSLOduration=10.916118413 podStartE2EDuration="10.916118413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:00.450048672 +0000 UTC m=+1740.778460058" watchObservedRunningTime="2026-10-30 19:06:09.916118413 +0000 UTC m=+1750.244529799" Oct 30 19:06:09.919315 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:09.919286 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2"] Oct 30 19:06:09.926619 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:09.926585 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:09.932511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:09.932481 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2"] Oct 30 19:06:10.004575 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.004527 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-dcd326-openshift-default-dockercfg-hmw94\"" Oct 30 19:06:10.041437 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.041404 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-token\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.041659 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.041448 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-data\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.041659 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.041541 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-credential-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.041772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.041745 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-envoy\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.041819 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.041803 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-podinfo\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.041858 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.041835 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edfaee8-04db-4f64-bdec-efe3df8a5439-istiod-ca-cert\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.041903 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.041874 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-certs\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.042035 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.042015 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bqdg\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-kube-api-access-8bqdg\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.042117 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.042101 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.143641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.143581 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-certs\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.143864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.143675 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqdg\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-kube-api-access-8bqdg\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.143864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.143722 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.143864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.143796 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-token\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.143864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.143816 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-data\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.143864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.143845 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-credential-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.144134 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.143940 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-envoy\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.144134 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.143979 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-podinfo\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.144134 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.144003 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edfaee8-04db-4f64-bdec-efe3df8a5439-istiod-ca-cert\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.144134 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.144055 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-certs\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.144307 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.144151 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.144307 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.144200 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-data\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.144307 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.144274 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-credential-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.144887 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.144690 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edfaee8-04db-4f64-bdec-efe3df8a5439-istiod-ca-cert\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.146446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.146424 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-envoy\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.146669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.146649 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-podinfo\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.177027 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.176952 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-token\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.177503 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.177475 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqdg\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-kube-api-access-8bqdg\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.251037 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.250999 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:10.549915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.549890 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2"] Oct 30 19:06:10.551577 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:06:10.551553 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edfaee8_04db_4f64_bdec_efe3df8a5439.slice/crio-317e767913b40f528d52c2022bb9ba1caffb438a3d77e09903952ad04160d504 WatchSource:0}: Error finding container 317e767913b40f528d52c2022bb9ba1caffb438a3d77e09903952ad04160d504: Status 404 returned error can't find the container with id 317e767913b40f528d52c2022bb9ba1caffb438a3d77e09903952ad04160d504 Oct 30 19:06:10.554159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.554127 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:06:10.554234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.554196 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:06:10.554234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:10.554225 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:06:11.425635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:11.423958 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:06:11.425635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:11.424641 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:06:11.489484 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:11.489424 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" event={"ID":"3edfaee8-04db-4f64-bdec-efe3df8a5439","Type":"ContainerStarted","Data":"b6277f4b41af3d2f8f0fd2eab8c6599a0d5cdd32ccbc314c84f28465d3f36434"} Oct 30 19:06:11.489746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:11.489678 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" event={"ID":"3edfaee8-04db-4f64-bdec-efe3df8a5439","Type":"ContainerStarted","Data":"317e767913b40f528d52c2022bb9ba1caffb438a3d77e09903952ad04160d504"} Oct 30 19:06:11.529093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:11.529028 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" podStartSLOduration=2.52900847 podStartE2EDuration="2.52900847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:11.51543585 +0000 UTC m=+1751.843847246" watchObservedRunningTime="2026-10-30 19:06:11.52900847 +0000 UTC m=+1751.857419857" Oct 30 19:06:12.251974 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:12.251929 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:12.254013 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:12.253979 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:06:12.254185 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:12.254043 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 19:06:13.252060 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:13.252016 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:06:13.252470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:13.252096 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 19:06:14.251980 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:14.251942 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:06:14.252140 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:14.252005 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 19:06:15.252247 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:15.252212 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:06:15.256749 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:15.255727 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 19:06:16.252159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:16.252117 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:06:16.252359 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:16.252180 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 19:06:17.008516 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:17.008472 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:06:17.265531 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:17.265438 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:17.266049 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:17.265949 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:17.267036 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:17.267013 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:06:22.494333 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.494297 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79"] Oct 30 19:06:22.499288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.499256 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.506681 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.506647 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-baa504-openshift-default-dockercfg-tbb42\"" Oct 30 19:06:22.508480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.508448 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79"] Oct 30 19:06:22.591684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.591643 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-envoy\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.591910 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.591725 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-data\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.591910 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.591817 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-token\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.591910 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.591862 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqjrw\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-kube-api-access-rqjrw\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.592053 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.591952 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-certs\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.592053 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.592032 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.592143 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.592071 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-podinfo\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.592189 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.592170 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16fb92f-e2df-4da5-b636-ea5f677da940-istiod-ca-cert\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.592224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.592196 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-credential-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694009 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.693959 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-certs\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694231 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694048 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694231 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694102 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-podinfo\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694231 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694165 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16fb92f-e2df-4da5-b636-ea5f677da940-istiod-ca-cert\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694231 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694201 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-credential-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694378 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-envoy\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694399 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-certs\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694426 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-data\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694461 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694474 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-token\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694802 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694564 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqjrw\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-kube-api-access-rqjrw\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694802 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694570 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-credential-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694901 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694823 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-data\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.694978 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.694960 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16fb92f-e2df-4da5-b636-ea5f677da940-istiod-ca-cert\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.696777 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.696753 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-envoy\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.696905 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.696872 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-podinfo\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.702953 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.702926 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-token\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.703199 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.703177 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqjrw\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-kube-api-access-rqjrw\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:22.822322 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:22.822210 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:23.132811 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:23.132779 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79"] Oct 30 19:06:23.137792 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:06:23.136728 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16fb92f_e2df_4da5_b636_ea5f677da940.slice/crio-ca9dda303431a651ca321ec5729bbeee5ad43c23dc51b3a3328c4df240a7d8ec WatchSource:0}: Error finding container ca9dda303431a651ca321ec5729bbeee5ad43c23dc51b3a3328c4df240a7d8ec: Status 404 returned error can't find the container with id ca9dda303431a651ca321ec5729bbeee5ad43c23dc51b3a3328c4df240a7d8ec Oct 30 19:06:23.141687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:23.141650 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:06:23.141796 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:23.141730 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:06:23.141796 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:23.141761 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:06:23.607233 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:23.607196 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" event={"ID":"c16fb92f-e2df-4da5-b636-ea5f677da940","Type":"ContainerStarted","Data":"c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c"} Oct 30 19:06:23.607686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:23.607278 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" event={"ID":"c16fb92f-e2df-4da5-b636-ea5f677da940","Type":"ContainerStarted","Data":"ca9dda303431a651ca321ec5729bbeee5ad43c23dc51b3a3328c4df240a7d8ec"} Oct 30 19:06:23.632413 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:23.632354 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" podStartSLOduration=1.632337873 podStartE2EDuration="1.632337873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:23.629950523 +0000 UTC m=+1763.958361909" watchObservedRunningTime="2026-10-30 19:06:23.632337873 +0000 UTC m=+1763.960749292" Oct 30 19:06:23.822879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:23.822841 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:24.824183 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:24.824148 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:06:24.824641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:24.824202 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 19:06:25.823382 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:25.823340 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:06:25.823548 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:25.823411 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 19:06:26.823254 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:26.823204 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:06:26.823805 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:26.823276 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 19:06:27.781570 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:06:27.781534 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:06:27.823369 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:27.823326 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:06:27.823846 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:27.823390 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 19:06:28.823108 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:28.823063 5959 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:06:28.823322 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:28.823123 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 19:06:30.016678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:30.016630 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:30.017979 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:30.017950 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:30.018802 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:30.018780 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:06:46.544319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.544281 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq"] Oct 30 19:06:46.551076 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.551045 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.552188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.552153 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-metrics-certs\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.552324 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.552275 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da26a2-90ab-43e0-9476-bcb1620532b8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.552588 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.552460 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8q68\" (UniqueName: \"kubernetes.io/projected/d0da26a2-90ab-43e0-9476-bcb1620532b8-kube-api-access-f8q68\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.552588 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.552517 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.559745 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.555786 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ca2cd5-serving-cert\"" Oct 30 19:06:46.562498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.561834 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq"] Oct 30 19:06:46.654760 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.654696 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8q68\" (UniqueName: \"kubernetes.io/projected/d0da26a2-90ab-43e0-9476-bcb1620532b8-kube-api-access-f8q68\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.656640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.655483 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.656640 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:06:46.655644 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-serving-cert: secret "maas-api-e2e-extmod-a-ca2cd5-serving-cert" not found Oct 30 19:06:46.656640 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:06:46.655912 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls podName:d0da26a2-90ab-43e0-9476-bcb1620532b8 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:47.15587014 +0000 UTC m=+1787.484281519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls") pod "maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" (UID: "d0da26a2-90ab-43e0-9476-bcb1620532b8") : secret "maas-api-e2e-extmod-a-ca2cd5-serving-cert" not found Oct 30 19:06:46.668803 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.668752 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-metrics-certs\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.669372 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.669338 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da26a2-90ab-43e0-9476-bcb1620532b8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.670728 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.670680 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da26a2-90ab-43e0-9476-bcb1620532b8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.679721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.679193 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-metrics-certs\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:46.680791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:46.680765 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8q68\" (UniqueName: \"kubernetes.io/projected/d0da26a2-90ab-43e0-9476-bcb1620532b8-kube-api-access-f8q68\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:47.178943 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:47.178904 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:47.182309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:47.182279 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:47.482214 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:47.482120 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:47.865713 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:47.865683 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq"] Oct 30 19:06:47.868961 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:06:47.868181 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0da26a2_90ab_43e0_9476_bcb1620532b8.slice/crio-ea2de3d48ee2b5b676049cf7eb9f5438a39ce4be989bfa6254fadfdfc778610e WatchSource:0}: Error finding container ea2de3d48ee2b5b676049cf7eb9f5438a39ce4be989bfa6254fadfdfc778610e: Status 404 returned error can't find the container with id ea2de3d48ee2b5b676049cf7eb9f5438a39ce4be989bfa6254fadfdfc778610e Oct 30 19:06:48.867661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:48.867593 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" event={"ID":"d0da26a2-90ab-43e0-9476-bcb1620532b8","Type":"ContainerStarted","Data":"0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc"} Oct 30 19:06:48.868210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:48.867694 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" event={"ID":"d0da26a2-90ab-43e0-9476-bcb1620532b8","Type":"ContainerStarted","Data":"ea2de3d48ee2b5b676049cf7eb9f5438a39ce4be989bfa6254fadfdfc778610e"} Oct 30 19:06:48.868210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:48.867987 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:51.652468 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:51.652411 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" podStartSLOduration=5.195862369 podStartE2EDuration="5.652392846s" totalImagesPullingTime="456.530477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:48.913355834 +0000 UTC m=+1789.241767219" watchObservedRunningTime="2026-10-30 19:06:51.652392846 +0000 UTC m=+1791.980804230" Oct 30 19:06:51.654768 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:51.654745 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj"] Oct 30 19:06:51.659635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:51.659586 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:51.675889 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:51.675861 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d6f956-dockercfg-9kfp5\"" Oct 30 19:06:51.677347 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:51.676700 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d6f956\"" Oct 30 19:06:51.682678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:51.682642 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj"] Oct 30 19:06:51.928304 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:51.928197 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q49v\" (UniqueName: \"kubernetes.io/projected/f126e5dc-9d83-4e52-8c42-ca8390a07330-kube-api-access-5q49v\") pod \"payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj\" (UID: \"f126e5dc-9d83-4e52-8c42-ca8390a07330\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:51.928655 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:51.928628 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f126e5dc-9d83-4e52-8c42-ca8390a07330-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj\" (UID: \"f126e5dc-9d83-4e52-8c42-ca8390a07330\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:52.030112 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.030076 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f126e5dc-9d83-4e52-8c42-ca8390a07330-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj\" (UID: \"f126e5dc-9d83-4e52-8c42-ca8390a07330\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:52.030347 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.030218 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q49v\" (UniqueName: \"kubernetes.io/projected/f126e5dc-9d83-4e52-8c42-ca8390a07330-kube-api-access-5q49v\") pod \"payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj\" (UID: \"f126e5dc-9d83-4e52-8c42-ca8390a07330\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:52.030958 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.030936 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f126e5dc-9d83-4e52-8c42-ca8390a07330-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj\" (UID: \"f126e5dc-9d83-4e52-8c42-ca8390a07330\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:52.040988 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.040951 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q49v\" (UniqueName: \"kubernetes.io/projected/f126e5dc-9d83-4e52-8c42-ca8390a07330-kube-api-access-5q49v\") pod \"payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj\" (UID: \"f126e5dc-9d83-4e52-8c42-ca8390a07330\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:52.292301 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.292209 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:52.603148 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:06:52.603115 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf126e5dc_9d83_4e52_8c42_ca8390a07330.slice/crio-83a40c0ae4f76e965b725138b39c3f63c49e091c7b3fccbda72bb2c516e3a57c WatchSource:0}: Error finding container 83a40c0ae4f76e965b725138b39c3f63c49e091c7b3fccbda72bb2c516e3a57c: Status 404 returned error can't find the container with id 83a40c0ae4f76e965b725138b39c3f63c49e091c7b3fccbda72bb2c516e3a57c Oct 30 19:06:52.603273 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.603188 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj"] Oct 30 19:06:52.943785 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.943745 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" event={"ID":"f126e5dc-9d83-4e52-8c42-ca8390a07330","Type":"ContainerStarted","Data":"c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69"} Oct 30 19:06:52.944228 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.943812 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" event={"ID":"f126e5dc-9d83-4e52-8c42-ca8390a07330","Type":"ContainerStarted","Data":"83a40c0ae4f76e965b725138b39c3f63c49e091c7b3fccbda72bb2c516e3a57c"} Oct 30 19:06:52.944228 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.943839 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:06:52.965321 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:52.965267 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" podStartSLOduration=1.965250162 podStartE2EDuration="1.965250162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:52.9627511 +0000 UTC m=+1793.291162506" watchObservedRunningTime="2026-10-30 19:06:52.965250162 +0000 UTC m=+1793.293661548" Oct 30 19:06:54.881391 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:06:54.881358 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:06:57.784895 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:06:57.784853 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:07:03.956593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:03.956563 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:07:04.861525 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:04.861474 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg"] Oct 30 19:07:04.869128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:04.869078 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:04.875326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:04.875274 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg"] Oct 30 19:07:04.878209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:04.878174 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d6f956-serving-cert\"" Oct 30 19:07:04.963171 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:04.963139 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wk9\" (UniqueName: \"kubernetes.io/projected/89938e5d-7f44-484a-ab58-355bc498e717-kube-api-access-d2wk9\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:04.963653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:04.963337 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89938e5d-7f44-484a-ab58-355bc498e717-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:04.963653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:04.963391 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:04.963801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:04.963696 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.070130 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.070070 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89938e5d-7f44-484a-ab58-355bc498e717-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.085660 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.079109 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.085660 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.079449 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.085660 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.079663 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wk9\" (UniqueName: \"kubernetes.io/projected/89938e5d-7f44-484a-ab58-355bc498e717-kube-api-access-d2wk9\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.085660 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.080756 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89938e5d-7f44-484a-ab58-355bc498e717-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.089308 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.089276 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.090298 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.090273 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wk9\" (UniqueName: \"kubernetes.io/projected/89938e5d-7f44-484a-ab58-355bc498e717-kube-api-access-d2wk9\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.091236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.091214 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.206318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.206274 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:05.509423 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:05.509397 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg"] Oct 30 19:07:05.512152 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:07:05.512116 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89938e5d_7f44_484a_ab58_355bc498e717.slice/crio-f9c716e0239050db553f62e46247975c0e662ba068154686415db7b65aa467b1 WatchSource:0}: Error finding container f9c716e0239050db553f62e46247975c0e662ba068154686415db7b65aa467b1: Status 404 returned error can't find the container with id f9c716e0239050db553f62e46247975c0e662ba068154686415db7b65aa467b1 Oct 30 19:07:06.103144 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:06.103102 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" event={"ID":"89938e5d-7f44-484a-ab58-355bc498e717","Type":"ContainerStarted","Data":"f23c664469fc019859231feb87220a10ec2d1f2cc28a0ea691ea324403548306"} Oct 30 19:07:06.103598 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:06.103172 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" event={"ID":"89938e5d-7f44-484a-ab58-355bc498e717","Type":"ContainerStarted","Data":"f9c716e0239050db553f62e46247975c0e662ba068154686415db7b65aa467b1"} Oct 30 19:07:06.103598 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:06.103467 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:06.124303 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:06.124245 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" podStartSLOduration=1.708905327 podStartE2EDuration="2.124228973s" totalImagesPullingTime="415.323646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:06.122941841 +0000 UTC m=+1806.451353253" watchObservedRunningTime="2026-10-30 19:07:06.124228973 +0000 UTC m=+1806.452640358" Oct 30 19:07:06.938677 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:06.936075 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq"] Oct 30 19:07:06.981633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:06.956743 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:06.984465 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:06.983421 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-2d1c71-openshift-default-dockercfg-vq6d2\"" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045169 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-data\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045294 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-envoy\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045431 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istiod-ca-cert\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045476 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-token\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045529 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045581 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-credential-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045684 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6fpb\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-kube-api-access-s6fpb\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045760 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-podinfo\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.080694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.045787 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-certs\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.145619 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.145572 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq"] Oct 30 19:07:07.148357 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148306 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-podinfo\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.148537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148361 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-certs\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.148537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148426 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-data\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.148537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148530 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-envoy\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.148726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148698 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istiod-ca-cert\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.148784 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148747 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-token\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.148833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148801 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.148888 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148853 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-credential-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.148978 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.148935 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6fpb\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-kube-api-access-s6fpb\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.151332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.151281 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istiod-ca-cert\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.151687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.151649 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-certs\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.152092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.152051 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-data\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.152822 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.152650 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.153727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.153702 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-credential-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.161224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.161192 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-podinfo\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.166366 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.165280 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-envoy\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.169332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.169256 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6fpb\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-kube-api-access-s6fpb\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.180785 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.180748 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-token\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.461686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.461588 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:07.758977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.758795 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq"] Oct 30 19:07:07.759234 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:07:07.759202 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eaf6ddb_5c3f_43dc_b49f_d23e1bb17408.slice/crio-ad47bee22de2ce5875fc0c008970fbc1d9304bb83400d0a5d4945e604437fa82 WatchSource:0}: Error finding container ad47bee22de2ce5875fc0c008970fbc1d9304bb83400d0a5d4945e604437fa82: Status 404 returned error can't find the container with id ad47bee22de2ce5875fc0c008970fbc1d9304bb83400d0a5d4945e604437fa82 Oct 30 19:07:07.762242 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.762203 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:07:07.762356 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.762273 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:07:07.762356 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:07.762303 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:07:08.176858 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:08.176819 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" event={"ID":"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408","Type":"ContainerStarted","Data":"1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58"} Oct 30 19:07:08.177340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:08.176876 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" event={"ID":"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408","Type":"ContainerStarted","Data":"ad47bee22de2ce5875fc0c008970fbc1d9304bb83400d0a5d4945e604437fa82"} Oct 30 19:07:08.216665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:08.216582 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" podStartSLOduration=2.216560155 podStartE2EDuration="2.216560155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:08.204429591 +0000 UTC m=+1808.532840978" watchObservedRunningTime="2026-10-30 19:07:08.216560155 +0000 UTC m=+1808.544971542" Oct 30 19:07:08.462726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:08.462624 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:08.464141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:08.464115 5959 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:07:08.464240 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:08.464176 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 19:07:09.462302 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:09.462265 5959 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:07:09.462774 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:09.462326 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 19:07:10.462361 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:10.462267 5959 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:07:10.462796 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:10.462351 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 19:07:11.462524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:11.462486 5959 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:07:11.462967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:11.462541 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 19:07:12.167536 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:12.167497 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:07:12.462909 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:12.462821 5959 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:07:12.462909 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:12.462887 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 19:07:13.462680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:13.462651 5959 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:07:13.462917 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:13.462894 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 19:07:14.490728 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:14.490690 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:14.491259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:14.491112 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:14.492257 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:14.492237 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:07:21.391026 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:21.390984 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5"] Oct 30 19:07:23.985384 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:23.985347 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:07:27.796283 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:07:27.796227 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:07:32.758775 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.758738 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj"] Oct 30 19:07:32.763686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.763666 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:32.767681 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.767655 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4a477d\"" Oct 30 19:07:32.768318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.768300 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4a477d-dockercfg-7nwph\"" Oct 30 19:07:32.772593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.772567 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj"] Oct 30 19:07:32.806291 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.806256 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n"] Oct 30 19:07:32.812732 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.812693 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:32.819045 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.819013 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n"] Oct 30 19:07:32.876630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.872790 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj\" (UID: \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:32.876630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:32.873010 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrcx\" (UniqueName: \"kubernetes.io/projected/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-kube-api-access-ggrcx\") pod \"payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj\" (UID: \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:33.025298 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.025112 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj\" (UID: \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:33.025936 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.025889 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrcx\" (UniqueName: \"kubernetes.io/projected/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-kube-api-access-ggrcx\") pod \"payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj\" (UID: \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:33.034671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.034431 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj\" (UID: \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:33.093505 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.093469 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrcx\" (UniqueName: \"kubernetes.io/projected/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-kube-api-access-ggrcx\") pod \"payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj\" (UID: \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:33.129172 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.129118 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n\" (UID: \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:33.129384 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.129281 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxlg\" (UniqueName: \"kubernetes.io/projected/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-kube-api-access-znxlg\") pod \"payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n\" (UID: \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:33.231452 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.231411 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n\" (UID: \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:33.231702 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.231481 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxlg\" (UniqueName: \"kubernetes.io/projected/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-kube-api-access-znxlg\") pod \"payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n\" (UID: \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:33.232398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.232366 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n\" (UID: \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:33.240549 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.240518 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxlg\" (UniqueName: \"kubernetes.io/projected/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-kube-api-access-znxlg\") pod \"payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n\" (UID: \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:33.384947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.384528 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:33.435010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.434969 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:33.679691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.679660 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj"] Oct 30 19:07:33.681079 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:07:33.681053 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98f9d6d_0773_4ae0_9a6b_b9aa7c880a07.slice/crio-2f650b0dd91f70885371dfedbf422f4fbefd0afa6bfc4144a24745b43031c5d3 WatchSource:0}: Error finding container 2f650b0dd91f70885371dfedbf422f4fbefd0afa6bfc4144a24745b43031c5d3: Status 404 returned error can't find the container with id 2f650b0dd91f70885371dfedbf422f4fbefd0afa6bfc4144a24745b43031c5d3 Oct 30 19:07:33.761882 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:33.760563 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n"] Oct 30 19:07:33.773191 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:07:33.773156 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f87578_e8f2_4a8a_9e6b_0525fd6c45db.slice/crio-6fdce402dcedbc6d5e09520174edaa620301b6e777ab0615bf985218ab5ba389 WatchSource:0}: Error finding container 6fdce402dcedbc6d5e09520174edaa620301b6e777ab0615bf985218ab5ba389: Status 404 returned error can't find the container with id 6fdce402dcedbc6d5e09520174edaa620301b6e777ab0615bf985218ab5ba389 Oct 30 19:07:34.509685 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:34.509644 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" event={"ID":"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07","Type":"ContainerStarted","Data":"33eda19fdb078aec365aa12b580b264237cafd8922e2c9bc665b257923a73d41"} Oct 30 19:07:34.509900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:34.509705 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" event={"ID":"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07","Type":"ContainerStarted","Data":"2f650b0dd91f70885371dfedbf422f4fbefd0afa6bfc4144a24745b43031c5d3"} Oct 30 19:07:34.509900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:34.509756 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:34.511812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:34.511773 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" event={"ID":"74f87578-e8f2-4a8a-9e6b-0525fd6c45db","Type":"ContainerStarted","Data":"fd9b34ac8a7794d89bd360cb45d41c0463e7322ba711e58f49f7fd2402e8291a"} Oct 30 19:07:34.512025 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:34.511838 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" event={"ID":"74f87578-e8f2-4a8a-9e6b-0525fd6c45db","Type":"ContainerStarted","Data":"6fdce402dcedbc6d5e09520174edaa620301b6e777ab0615bf985218ab5ba389"} Oct 30 19:07:34.512025 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:34.511872 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:34.531067 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:34.531018 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" podStartSLOduration=2.531002504 podStartE2EDuration="2.531002504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:34.530424229 +0000 UTC m=+1834.858835613" watchObservedRunningTime="2026-10-30 19:07:34.531002504 +0000 UTC m=+1834.859413889" Oct 30 19:07:34.553583 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:34.553527 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" podStartSLOduration=2.553512195 podStartE2EDuration="2.553512195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:34.549592562 +0000 UTC m=+1834.878003974" watchObservedRunningTime="2026-10-30 19:07:34.553512195 +0000 UTC m=+1834.881923594" Oct 30 19:07:36.429845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.429772 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" podUID="73b2276f-8831-4420-8355-44363744c018" containerName="main" containerID="cri-o://7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc" gracePeriod=45 Oct 30 19:07:36.753811 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.753775 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:07:36.792641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.792584 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-home\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-home\") pod \"73b2276f-8831-4420-8355-44363744c018\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " Oct 30 19:07:36.792860 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.792658 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73b2276f-8831-4420-8355-44363744c018-tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2276f-8831-4420-8355-44363744c018-tls-certs\") pod \"73b2276f-8831-4420-8355-44363744c018\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " Oct 30 19:07:36.792860 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.792685 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-dshm\") pod \"73b2276f-8831-4420-8355-44363744c018\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " Oct 30 19:07:36.792860 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.792812 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-model-cache\") pod \"73b2276f-8831-4420-8355-44363744c018\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " Oct 30 19:07:36.793037 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.792897 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b2276f-8831-4420-8355-44363744c018-kube-api-access-jjd8z\" (UniqueName: \"kubernetes.io/projected/73b2276f-8831-4420-8355-44363744c018-kube-api-access-jjd8z\") pod \"73b2276f-8831-4420-8355-44363744c018\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " Oct 30 19:07:36.793037 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.792911 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-home" pod "73b2276f-8831-4420-8355-44363744c018" (UID: "73b2276f-8831-4420-8355-44363744c018"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:36.793154 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.793062 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-tmp-dir\") pod \"73b2276f-8831-4420-8355-44363744c018\" (UID: \"73b2276f-8831-4420-8355-44363744c018\") " Oct 30 19:07:36.793663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.793629 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-model-cache" pod "73b2276f-8831-4420-8355-44363744c018" (UID: "73b2276f-8831-4420-8355-44363744c018"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:36.793797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.793692 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-tmp-dir" pod "73b2276f-8831-4420-8355-44363744c018" (UID: "73b2276f-8831-4420-8355-44363744c018"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:36.794043 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.794016 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:07:36.794109 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.794049 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:07:36.794109 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.794064 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:07:36.796231 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.796197 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b2276f-8831-4420-8355-44363744c018-kube-api-access-jjd8z" pod "73b2276f-8831-4420-8355-44363744c018" (UID: "73b2276f-8831-4420-8355-44363744c018"). InnerVolumeSpecName "kube-api-access-jjd8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:36.796334 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.796287 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-dshm" pod "73b2276f-8831-4420-8355-44363744c018" (UID: "73b2276f-8831-4420-8355-44363744c018"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:36.798410 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.798386 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b2276f-8831-4420-8355-44363744c018-tls-certs" pod "73b2276f-8831-4420-8355-44363744c018" (UID: "73b2276f-8831-4420-8355-44363744c018"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:36.895475 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.895440 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2276f-8831-4420-8355-44363744c018-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:07:36.895475 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.895473 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2276f-8831-4420-8355-44363744c018-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:07:36.895475 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:36.895483 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjd8z\" (UniqueName: \"kubernetes.io/projected/73b2276f-8831-4420-8355-44363744c018-kube-api-access-jjd8z\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:07:37.536421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.536388 5959 generic.go:356] "Generic (PLEG): container finished" podID="73b2276f-8831-4420-8355-44363744c018" containerID="7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc" exitCode=0 Oct 30 19:07:37.536923 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.536428 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" event={"ID":"73b2276f-8831-4420-8355-44363744c018","Type":"ContainerDied","Data":"7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc"} Oct 30 19:07:37.536923 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.536438 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" Oct 30 19:07:37.536923 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.536456 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5" event={"ID":"73b2276f-8831-4420-8355-44363744c018","Type":"ContainerDied","Data":"92371c096b823ed02ee0bdbe3ed5fb1df51cc1f9a28cfc8fe10585af9c7bcbf3"} Oct 30 19:07:37.536923 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.536473 5959 scope.go:122] "RemoveContainer" containerID="7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc" Oct 30 19:07:37.550310 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.550283 5959 scope.go:122] "RemoveContainer" containerID="7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc" Oct 30 19:07:37.550743 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:07:37.550721 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc\": container with ID starting with 7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc not found: ID does not exist" containerID="7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc" Oct 30 19:07:37.550831 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.550753 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc"} err="failed to get container status \"7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc\": rpc error: code = NotFound desc = could not find container \"7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc\": container with ID starting with 7302f46b571fd03abac6973e18e4ab0c43fe74185aa1c922f109e1e3ec3775bc not found: ID does not exist" Oct 30 19:07:37.563226 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.563190 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5"] Oct 30 19:07:37.566663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.566621 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2d1c71/auth-test-model-2d1c71-kserve-5c8b96ccc-nx7l5"] Oct 30 19:07:37.801059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:37.800969 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b2276f-8831-4420-8355-44363744c018" path="/var/lib/kubelet/pods/73b2276f-8831-4420-8355-44363744c018/volumes" Oct 30 19:07:45.523039 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:45.522400 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:07:45.528011 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:45.527969 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:07:47.783222 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.783186 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp"] Oct 30 19:07:47.784279 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.784256 5959 cpu_manager.go:414] "containerMap: removing container" podUID="73b2276f-8831-4420-8355-44363744c018" containerName="main" Oct 30 19:07:47.784279 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.784280 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b2276f-8831-4420-8355-44363744c018" containerName="main" Oct 30 19:07:47.784423 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.784406 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b2276f-8831-4420-8355-44363744c018" containerName="main" Oct 30 19:07:47.790148 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.790114 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.802699 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.802671 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4a477d-serving-cert\"" Oct 30 19:07:47.811318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.811287 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp"] Oct 30 19:07:47.892269 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.892233 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76c6k\" (UniqueName: \"kubernetes.io/projected/a1c0c547-d980-42ee-8b82-411a2bf05e64-kube-api-access-76c6k\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.892620 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.892573 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c0c547-d980-42ee-8b82-411a2bf05e64-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.892715 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.892695 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.892815 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.892799 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-metrics-certs\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.995786 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.995749 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c0c547-d980-42ee-8b82-411a2bf05e64-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.995999 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.995830 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.995999 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.995899 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-metrics-certs\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.995999 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:07:47.995982 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-serving-cert: secret "maas-api-e2e-extmod-b-4a477d-serving-cert" not found Oct 30 19:07:47.996194 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.996044 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76c6k\" (UniqueName: \"kubernetes.io/projected/a1c0c547-d980-42ee-8b82-411a2bf05e64-kube-api-access-76c6k\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.996194 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:07:47.996062 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls podName:a1c0c547-d980-42ee-8b82-411a2bf05e64 nodeName:}" failed. No retries permitted until 2026-10-30 19:07:48.496039339 +0000 UTC m=+1848.824450705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls") pod "maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" (UID: "a1c0c547-d980-42ee-8b82-411a2bf05e64") : secret "maas-api-e2e-extmod-b-4a477d-serving-cert" not found Oct 30 19:07:47.996822 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.996592 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c0c547-d980-42ee-8b82-411a2bf05e64-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:47.998685 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:47.998664 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-metrics-certs\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:48.005037 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:48.005013 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76c6k\" (UniqueName: \"kubernetes.io/projected/a1c0c547-d980-42ee-8b82-411a2bf05e64-kube-api-access-76c6k\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:48.510632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:48.507299 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:48.518640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:48.517250 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4a477d-8545544489-d82bp\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:48.726970 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:48.726923 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:49.014102 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:49.014064 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp"] Oct 30 19:07:49.015768 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:07:49.015737 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c0c547_d980_42ee_8b82_411a2bf05e64.slice/crio-1bf213fbcc480cfb4dfc3beaf6268377b2e6174f5095f8052c96c7c7e502c7eb WatchSource:0}: Error finding container 1bf213fbcc480cfb4dfc3beaf6268377b2e6174f5095f8052c96c7c7e502c7eb: Status 404 returned error can't find the container with id 1bf213fbcc480cfb4dfc3beaf6268377b2e6174f5095f8052c96c7c7e502c7eb Oct 30 19:07:49.017264 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:49.017244 5959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:07:49.844273 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:49.844170 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" event={"ID":"a1c0c547-d980-42ee-8b82-411a2bf05e64","Type":"ContainerStarted","Data":"fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2"} Oct 30 19:07:49.844273 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:49.844226 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" event={"ID":"a1c0c547-d980-42ee-8b82-411a2bf05e64","Type":"ContainerStarted","Data":"1bf213fbcc480cfb4dfc3beaf6268377b2e6174f5095f8052c96c7c7e502c7eb"} Oct 30 19:07:49.844515 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:49.844304 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:49.865249 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:49.865183 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" podStartSLOduration=2.405878038 podStartE2EDuration="2.865168678s" totalImagesPullingTime="459.29064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:49.863102828 +0000 UTC m=+1850.191514213" watchObservedRunningTime="2026-10-30 19:07:49.865168678 +0000 UTC m=+1850.193580173" Oct 30 19:07:50.632001 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.631959 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7"] Oct 30 19:07:50.637232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.637198 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:50.642954 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.642927 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e67d95-dockercfg-lfx22\"" Oct 30 19:07:50.645356 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.645327 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7"] Oct 30 19:07:50.709873 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.709826 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e67d95\"" Oct 30 19:07:50.784414 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.768947 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmf8h\" (UniqueName: \"kubernetes.io/projected/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-kube-api-access-fmf8h\") pod \"payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7\" (UID: \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:50.784414 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.769311 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7\" (UID: \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:50.897683 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.896262 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmf8h\" (UniqueName: \"kubernetes.io/projected/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-kube-api-access-fmf8h\") pod \"payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7\" (UID: \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:50.897683 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.896413 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7\" (UID: \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:50.899099 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.899059 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7\" (UID: \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:50.916444 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.916410 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmf8h\" (UniqueName: \"kubernetes.io/projected/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-kube-api-access-fmf8h\") pod \"payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7\" (UID: \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:50.961443 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:50.961396 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:51.270238 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:51.270202 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7"] Oct 30 19:07:51.270581 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:07:51.270555 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7b40fc_9360_4a4b_baa0_02fcb9063ac0.slice/crio-c9606b95952f586cb7fb21b01e04f3b99e338a9d7aa131e56b42b17257f0856e WatchSource:0}: Error finding container c9606b95952f586cb7fb21b01e04f3b99e338a9d7aa131e56b42b17257f0856e: Status 404 returned error can't find the container with id c9606b95952f586cb7fb21b01e04f3b99e338a9d7aa131e56b42b17257f0856e Oct 30 19:07:51.891661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:51.891593 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" event={"ID":"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0","Type":"ContainerStarted","Data":"d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6"} Oct 30 19:07:51.892086 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:51.891676 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" event={"ID":"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0","Type":"ContainerStarted","Data":"c9606b95952f586cb7fb21b01e04f3b99e338a9d7aa131e56b42b17257f0856e"} Oct 30 19:07:51.892086 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:51.891805 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:07:51.914948 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:51.914888 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" podStartSLOduration=1.914868351 podStartE2EDuration="1.914868351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:51.912357162 +0000 UTC m=+1852.240768548" watchObservedRunningTime="2026-10-30 19:07:51.914868351 +0000 UTC m=+1852.243279741" Oct 30 19:07:55.891215 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:07:55.891180 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:07:57.783758 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:07:57.783722 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:08:02.942494 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:02.942458 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:08:03.311768 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.311664 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs"] Oct 30 19:08:03.352089 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.352045 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:03.353985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.353948 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs"] Oct 30 19:08:03.359092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.359066 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c164d6-dockercfg-5hndx\"" Oct 30 19:08:03.397634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.385843 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c164d6\"" Oct 30 19:08:03.524944 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.524906 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8bq\" (UniqueName: \"kubernetes.io/projected/3d484e54-c577-47a7-a127-33548ae935bf-kube-api-access-rn8bq\") pod \"payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs\" (UID: \"3d484e54-c577-47a7-a127-33548ae935bf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:03.525331 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.525308 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d484e54-c577-47a7-a127-33548ae935bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs\" (UID: \"3d484e54-c577-47a7-a127-33548ae935bf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:03.633344 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.633236 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d484e54-c577-47a7-a127-33548ae935bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs\" (UID: \"3d484e54-c577-47a7-a127-33548ae935bf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:03.634094 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.634054 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8bq\" (UniqueName: \"kubernetes.io/projected/3d484e54-c577-47a7-a127-33548ae935bf-kube-api-access-rn8bq\") pod \"payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs\" (UID: \"3d484e54-c577-47a7-a127-33548ae935bf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:03.637156 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.636592 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d484e54-c577-47a7-a127-33548ae935bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs\" (UID: \"3d484e54-c577-47a7-a127-33548ae935bf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:03.690787 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.688627 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8bq\" (UniqueName: \"kubernetes.io/projected/3d484e54-c577-47a7-a127-33548ae935bf-kube-api-access-rn8bq\") pod \"payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs\" (UID: \"3d484e54-c577-47a7-a127-33548ae935bf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:03.712730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:03.712694 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:04.049813 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:04.049777 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs"] Oct 30 19:08:04.052034 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:08:04.052004 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d484e54_c577_47a7_a127_33548ae935bf.slice/crio-37dc3ed50b684270d8c966699c01e7bbaf78129c5062838cbc2dbf0268fd3112 WatchSource:0}: Error finding container 37dc3ed50b684270d8c966699c01e7bbaf78129c5062838cbc2dbf0268fd3112: Status 404 returned error can't find the container with id 37dc3ed50b684270d8c966699c01e7bbaf78129c5062838cbc2dbf0268fd3112 Oct 30 19:08:05.053170 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:05.053129 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" event={"ID":"3d484e54-c577-47a7-a127-33548ae935bf","Type":"ContainerStarted","Data":"3c2b48e7b36412a348a762dfda3762f9b918376d890b0073d89d68e7dbbc597a"} Oct 30 19:08:05.053722 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:05.053192 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:05.053722 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:05.053217 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" event={"ID":"3d484e54-c577-47a7-a127-33548ae935bf","Type":"ContainerStarted","Data":"37dc3ed50b684270d8c966699c01e7bbaf78129c5062838cbc2dbf0268fd3112"} Oct 30 19:08:05.075812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:05.075758 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" podStartSLOduration=2.075738631 podStartE2EDuration="2.075738631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:05.073677587 +0000 UTC m=+1865.402088977" watchObservedRunningTime="2026-10-30 19:08:05.075738631 +0000 UTC m=+1865.404150017" Oct 30 19:08:14.920246 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:14.920205 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s"] Oct 30 19:08:14.920793 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:14.920478 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" containerID="cri-o://f566bc1d57c95bfac8ad96a95230550b27112ba690470168397aab5f4c5cd258" gracePeriod=30 Oct 30 19:08:15.593635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:15.576073 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk"] Oct 30 19:08:15.599385 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:15.599320 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" containerID="cri-o://95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981" gracePeriod=30 Oct 30 19:08:16.119851 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:16.119816 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:08:20.246891 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.246857 5959 generic.go:356] "Generic (PLEG): container finished" podID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerID="f566bc1d57c95bfac8ad96a95230550b27112ba690470168397aab5f4c5cd258" exitCode=0 Oct 30 19:08:20.247283 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.246925 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" event={"ID":"a58ad637-7f3a-49c6-93e4-97a9ea5c584b","Type":"ContainerDied","Data":"f566bc1d57c95bfac8ad96a95230550b27112ba690470168397aab5f4c5cd258"} Oct 30 19:08:20.247283 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.246980 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" event={"ID":"a58ad637-7f3a-49c6-93e4-97a9ea5c584b","Type":"ContainerDied","Data":"0dc46c1d9abd7f4b209d6b02b6198d6a5bf853b22b4dc39b6920954e3cae62c5"} Oct 30 19:08:20.247283 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.247000 5959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dc46c1d9abd7f4b209d6b02b6198d6a5bf853b22b4dc39b6920954e3cae62c5" Oct 30 19:08:20.255453 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.255425 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:08:20.415678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.415638 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-envoy\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.415874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.415703 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-podinfo\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.415874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.415811 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-socket\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.415874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.415860 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-data\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.416108 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.416087 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-certs\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.416216 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.416196 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-credential-socket\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.416276 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.416191 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-socket" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:20.416585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.416263 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-token\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-token\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.416585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.416339 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-kube-api-access-m6l88\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-kube-api-access-m6l88\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.416585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.416437 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istiod-ca-cert\") pod \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\" (UID: \"a58ad637-7f3a-49c6-93e4-97a9ea5c584b\") " Oct 30 19:08:20.417050 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.417024 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-data" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:20.417366 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.417044 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-certs" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:20.417546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.417524 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istiod-ca-cert" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:20.417855 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.417835 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.417988 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.417974 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.418125 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.418111 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.418257 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.418245 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.418375 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.417860 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-credential-socket" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:20.418493 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.418469 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-envoy" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:20.419704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.419674 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-podinfo" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:20.419821 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.419701 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-kube-api-access-m6l88" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "kube-api-access-m6l88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:20.420214 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.420195 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-token" pod "a58ad637-7f3a-49c6-93e4-97a9ea5c584b" (UID: "a58ad637-7f3a-49c6-93e4-97a9ea5c584b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:20.520062 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.519959 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.520062 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.519997 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.520062 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.520011 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6l88\" (UniqueName: \"kubernetes.io/projected/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-kube-api-access-m6l88\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.520062 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.520023 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.520062 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.520035 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58ad637-7f3a-49c6-93e4-97a9ea5c584b-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:20.911215 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:20.911182 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 19:08:21.051996 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.051717 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-envoy\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.052285 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.052261 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-credential-socket\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.052446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.052424 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-token\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-token\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.052542 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.052527 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-kube-api-access-cmq7v\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-kube-api-access-cmq7v\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.052703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.052670 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-podinfo\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.052781 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.052713 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istiod-ca-cert\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.052842 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.052821 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-certs\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.052902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.052870 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-data\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.052990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.052975 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-socket\") pod \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\" (UID: \"55815db4-66e6-42a3-aa1c-9e49fb47bb06\") " Oct 30 19:08:21.054262 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.054236 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-socket" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:21.054501 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.054476 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-credential-socket" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:21.056673 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.056639 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-certs" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:21.057240 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.057202 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istiod-ca-cert" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:21.057503 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.057477 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-token" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:21.060579 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.060552 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-podinfo" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:21.062010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.061988 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-data" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:21.064187 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.064140 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-envoy" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:21.064187 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.064159 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-kube-api-access-cmq7v" pod "55815db4-66e6-42a3-aa1c-9e49fb47bb06" (UID: "55815db4-66e6-42a3-aa1c-9e49fb47bb06"). InnerVolumeSpecName "kube-api-access-cmq7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:21.155106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155059 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmq7v\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-kube-api-access-cmq7v\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.155106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155098 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.155106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155109 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.155106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155118 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.155498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155126 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.155498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155134 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.155498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155142 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.155498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155150 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55815db4-66e6-42a3-aa1c-9e49fb47bb06-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.155498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.155158 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55815db4-66e6-42a3-aa1c-9e49fb47bb06-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:21.258042 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.258007 5959 generic.go:356] "Generic (PLEG): container finished" podID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerID="95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981" exitCode=0 Oct 30 19:08:21.258510 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.258084 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" event={"ID":"55815db4-66e6-42a3-aa1c-9e49fb47bb06","Type":"ContainerDied","Data":"95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981"} Oct 30 19:08:21.258510 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.258095 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" Oct 30 19:08:21.258510 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.258180 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk" event={"ID":"55815db4-66e6-42a3-aa1c-9e49fb47bb06","Type":"ContainerDied","Data":"4e1ea873813b4d254509ae5e054c99f80f1aeabb14b1fb5f6f950aab2bb20614"} Oct 30 19:08:21.258510 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.258209 5959 scope.go:122] "RemoveContainer" containerID="95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981" Oct 30 19:08:21.258510 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.258398 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s" Oct 30 19:08:21.273290 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.272343 5959 scope.go:122] "RemoveContainer" containerID="95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981" Oct 30 19:08:21.273290 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:08:21.272785 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981\": container with ID starting with 95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981 not found: ID does not exist" containerID="95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981" Oct 30 19:08:21.273290 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.272824 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981"} err="failed to get container status \"95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981\": rpc error: code = NotFound desc = could not find container \"95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981\": container with ID starting with 95574844cc4030d57b7f98473ff4baba3f1426b39d0203904c35fe9e088df981 not found: ID does not exist" Oct 30 19:08:21.290522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.290488 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s"] Oct 30 19:08:21.296145 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.296113 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-p4w5s"] Oct 30 19:08:21.307914 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.307101 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk"] Oct 30 19:08:21.311677 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.311646 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-ghbmk"] Oct 30 19:08:21.776462 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.776427 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568"] Oct 30 19:08:21.777360 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.777337 5959 cpu_manager.go:414] "containerMap: removing container" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" Oct 30 19:08:21.777360 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.777360 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" Oct 30 19:08:21.777552 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.777383 5959 cpu_manager.go:414] "containerMap: removing container" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" Oct 30 19:08:21.777552 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.777390 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" Oct 30 19:08:21.777728 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.777712 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" containerName="istio-proxy" Oct 30 19:08:21.777794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.777736 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" containerName="istio-proxy" Oct 30 19:08:21.783338 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.783315 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.793683 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.791686 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f3d6a5-openshift-default-dockercfg-8dmt5\"" Oct 30 19:08:21.815833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.811313 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55815db4-66e6-42a3-aa1c-9e49fb47bb06" path="/var/lib/kubelet/pods/55815db4-66e6-42a3-aa1c-9e49fb47bb06/volumes" Oct 30 19:08:21.815833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.812169 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58ad637-7f3a-49c6-93e4-97a9ea5c584b" path="/var/lib/kubelet/pods/a58ad637-7f3a-49c6-93e4-97a9ea5c584b/volumes" Oct 30 19:08:21.815833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.812646 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568"] Oct 30 19:08:21.815833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.812708 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4"] Oct 30 19:08:21.823363 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.823328 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4"] Oct 30 19:08:21.823785 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.823760 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.864386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.864351 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.864589 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.864397 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.864589 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.864513 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.864589 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.864580 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.864794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.864704 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cae719-52c3-492a-aa66-c4db3a011fee-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.864794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.864772 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8563f5fd-0739-44cb-acbf-e98721991c05-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.864892 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.864803 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkwlw\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-kube-api-access-kkwlw\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.864951 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.864931 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.865045 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865028 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.865093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865065 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.865137 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865091 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.865231 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865215 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.865277 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865260 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8563f5fd-0739-44cb-acbf-e98721991c05-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.865383 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865357 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.865442 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865426 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtpb\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-kube-api-access-wgtpb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.865511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865494 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.865549 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865528 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.865811 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.865783 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.967735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.967700 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.967947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.967754 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtpb\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-kube-api-access-wgtpb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.967947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.967787 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.967947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.967803 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.967947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.967846 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.967947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.967909 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.967947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.967939 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.968224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.967988 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.968224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968048 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.968224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968091 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cae719-52c3-492a-aa66-c4db3a011fee-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.968224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968125 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8563f5fd-0739-44cb-acbf-e98721991c05-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.968224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968166 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkwlw\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-kube-api-access-kkwlw\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.968490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968244 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.968490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968310 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.968490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968319 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.968490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968338 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.968490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968389 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.968490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968394 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.968490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968420 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.968490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968447 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.969010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968545 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.969010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968581 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.969010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968589 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8563f5fd-0739-44cb-acbf-e98721991c05-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.969010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.968729 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.969252 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.969051 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.969252 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.969080 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cae719-52c3-492a-aa66-c4db3a011fee-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.969252 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.969093 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8563f5fd-0739-44cb-acbf-e98721991c05-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.969433 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.969316 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.970845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.970818 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.971108 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.971089 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.971531 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.971508 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.971857 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.971837 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8563f5fd-0739-44cb-acbf-e98721991c05-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.976163 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.976124 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:21.976292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.976197 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.976292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.976252 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkwlw\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-kube-api-access-kkwlw\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:21.976638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:21.976585 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtpb\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-kube-api-access-wgtpb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:22.104680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.104556 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:22.148098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.142939 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:22.432415 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.432387 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568"] Oct 30 19:08:22.433712 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:08:22.433685 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cae719_52c3_492a_aa66_c4db3a011fee.slice/crio-bfa1b99d9e346523f3e89310daebeeff7efa79318056f9ea164809bbfdbc836e WatchSource:0}: Error finding container bfa1b99d9e346523f3e89310daebeeff7efa79318056f9ea164809bbfdbc836e: Status 404 returned error can't find the container with id bfa1b99d9e346523f3e89310daebeeff7efa79318056f9ea164809bbfdbc836e Oct 30 19:08:22.436738 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.436708 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:22.436815 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.436775 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:22.436815 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.436806 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:22.498347 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.498314 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4"] Oct 30 19:08:22.503446 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:08:22.503401 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8563f5fd_0739_44cb_acbf_e98721991c05.slice/crio-0f649073be7c5b6d0b03fe5cad38be189b04f3c889db3bff00e42cbf7a55e854 WatchSource:0}: Error finding container 0f649073be7c5b6d0b03fe5cad38be189b04f3c889db3bff00e42cbf7a55e854: Status 404 returned error can't find the container with id 0f649073be7c5b6d0b03fe5cad38be189b04f3c889db3bff00e42cbf7a55e854 Oct 30 19:08:22.507110 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.507076 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:22.507221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.507168 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:22.507221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:22.507215 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:23.285559 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.285513 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" event={"ID":"e5cae719-52c3-492a-aa66-c4db3a011fee","Type":"ContainerStarted","Data":"c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5"} Oct 30 19:08:23.285790 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.285597 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" event={"ID":"e5cae719-52c3-492a-aa66-c4db3a011fee","Type":"ContainerStarted","Data":"bfa1b99d9e346523f3e89310daebeeff7efa79318056f9ea164809bbfdbc836e"} Oct 30 19:08:23.288327 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.288295 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" event={"ID":"8563f5fd-0739-44cb-acbf-e98721991c05","Type":"ContainerStarted","Data":"360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927"} Oct 30 19:08:23.288479 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.288349 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" event={"ID":"8563f5fd-0739-44cb-acbf-e98721991c05","Type":"ContainerStarted","Data":"0f649073be7c5b6d0b03fe5cad38be189b04f3c889db3bff00e42cbf7a55e854"} Oct 30 19:08:23.324633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.323992 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" podStartSLOduration=2.3239691430000002 podStartE2EDuration="2.323969143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:23.305893338 +0000 UTC m=+1883.634304715" watchObservedRunningTime="2026-10-30 19:08:23.323969143 +0000 UTC m=+1883.652380531" Oct 30 19:08:23.358146 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.358056 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" podStartSLOduration=2.358034774 podStartE2EDuration="2.358034774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:23.3494924 +0000 UTC m=+1883.677903798" watchObservedRunningTime="2026-10-30 19:08:23.358034774 +0000 UTC m=+1883.686446160" Oct 30 19:08:23.460148 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.460116 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h"] Oct 30 19:08:23.465313 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.465287 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.474442 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.474410 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e51c10-openshift-default-dockercfg-s9hfx\"" Oct 30 19:08:23.478281 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.478249 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h"] Oct 30 19:08:23.484461 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.484431 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm"] Oct 30 19:08:23.535691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.522828 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm"] Oct 30 19:08:23.535691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.522998 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.663666 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hpxn\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-kube-api-access-5hpxn\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.663815 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.663986 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.664047 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.664175 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.664263 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.664294 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.664403 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.664442 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665054 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665091 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665141 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qgc\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-kube-api-access-n6qgc\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665319 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665383 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665425 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.665669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665462 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.666686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665750 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.666686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.665789 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.768751 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.768170 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.769081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.769043 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.769534 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.769515 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.770079 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.770050 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.770398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.770350 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.770706 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.770683 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.771352 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.771333 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.772041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.771874 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.772304 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.772284 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hpxn\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-kube-api-access-5hpxn\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.772938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.772814 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.773145 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773098 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.773309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773290 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773396 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773416 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773508 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773561 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773661 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773689 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773727 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773754 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773823 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qgc\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-kube-api-access-n6qgc\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.773921 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.774305 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.774682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.774584 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.775368 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.774755 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.775368 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.774996 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.778489 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.776003 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.778489 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.778203 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.779291 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.778889 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.783647 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.781523 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.783647 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.782711 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.783647 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.783303 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hpxn\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-kube-api-access-5hpxn\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.783966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.783944 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.785957 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.785856 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.786707 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.786658 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:23.788224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.788205 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qgc\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-kube-api-access-n6qgc\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.824249 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.823955 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:23.893986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:23.893957 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:24.151005 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.149656 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:24.151005 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.149939 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:24.182697 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.182192 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:08:24.182697 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.182261 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:08:24.182697 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.182649 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:08:24.183087 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.182696 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:08:24.460026 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.459980 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm"] Oct 30 19:08:24.460931 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:08:24.460891 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab751835_0c83_473f_a8ff_b17e93a9a3a3.slice/crio-0669874a33b840182c0f9584a8892e9fae33a98d586edb3ee88f303ab483fe55 WatchSource:0}: Error finding container 0669874a33b840182c0f9584a8892e9fae33a98d586edb3ee88f303ab483fe55: Status 404 returned error can't find the container with id 0669874a33b840182c0f9584a8892e9fae33a98d586edb3ee88f303ab483fe55 Oct 30 19:08:24.465797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.465630 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:24.465967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.465938 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:24.466060 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.466044 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:24.487988 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.484174 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h"] Oct 30 19:08:24.571723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.571677 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:24.571817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.571791 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:24.571864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:24.571839 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:08:25.115640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.108668 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:08:25.115640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.108751 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:08:25.145636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.145143 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:08:25.145636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.145223 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:08:25.370688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.370034 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" event={"ID":"ab751835-0c83-473f-a8ff-b17e93a9a3a3","Type":"ContainerStarted","Data":"61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1"} Oct 30 19:08:25.370688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.370100 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" event={"ID":"ab751835-0c83-473f-a8ff-b17e93a9a3a3","Type":"ContainerStarted","Data":"0669874a33b840182c0f9584a8892e9fae33a98d586edb3ee88f303ab483fe55"} Oct 30 19:08:25.373629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.372921 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" event={"ID":"4cde4d2c-1782-41cc-9011-a7ac2ee0d079","Type":"ContainerStarted","Data":"f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16"} Oct 30 19:08:25.373629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.372980 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" event={"ID":"4cde4d2c-1782-41cc-9011-a7ac2ee0d079","Type":"ContainerStarted","Data":"1e9578103612d90d703c89efe27f2912e9f0ce9d26aacd68b2867b0892d9920b"} Oct 30 19:08:25.410723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.410657 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podStartSLOduration=2.410633165 podStartE2EDuration="2.410633165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:25.395829116 +0000 UTC m=+1885.724240500" watchObservedRunningTime="2026-10-30 19:08:25.410633165 +0000 UTC m=+1885.739044551" Oct 30 19:08:25.425746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.425686 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podStartSLOduration=2.425669763 podStartE2EDuration="2.425669763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:25.424112818 +0000 UTC m=+1885.752524204" watchObservedRunningTime="2026-10-30 19:08:25.425669763 +0000 UTC m=+1885.754081147" Oct 30 19:08:25.872630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.867147 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:08:25.872630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.867212 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:08:25.891713 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.883330 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:25.895898 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.895869 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:25.899628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.896175 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:08:25.899913 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:25.899878 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:08:26.115634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.115395 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:08:26.115634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.115460 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:08:26.175635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.163768 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:08:26.175635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.163839 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:08:26.841629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.841110 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:08:26.841629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.841173 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:08:26.870752 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.870703 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:26.897058 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.897017 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:08:26.898761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.898725 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:08:26.915753 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.915707 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:26.916418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.916022 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:26.963148 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:26.963103 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:27.038761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.038715 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:27.051631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.049698 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:27.075634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.055702 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:27.114693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.114016 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:08:27.114693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.114083 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:08:27.145633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.145015 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:08:27.145633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.145082 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:08:27.445810 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.445752 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cjqh2" podUID="193f879e-f190-4a8d-8b3b-dcf68accc4ef" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.25:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:27.521647 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.513896 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:27.610631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.609863 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 19:08:27.859823 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.859722 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:08:27.864734 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.864684 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:08:27.891833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.889713 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:27.891833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.890436 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:08:27.894123 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.894094 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:08:27.935633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.922814 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:08:27.936275 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.936234 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:08:27.938318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.922868 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:27.960633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.922898 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 19:08:27.960633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.958383 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:08:27.960633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.958448 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:08:27.964216 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.964176 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 19:08:27.964840 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:27.964817 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:08:28.002641 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:08:27.987885 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:08:28.111782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.075579 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:28.111782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.076013 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:28.111782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.094759 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:08:28.111782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.094840 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:08:28.181653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.180022 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:28.181653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.180425 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:08:28.202874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.182088 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:28.202874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.182565 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:08:28.215219 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.214107 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:08:28.215219 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.214170 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:08:28.343631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.338705 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:08:28.894071 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.894034 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:08:28.917030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.913892 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:08:28.917030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.913964 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:08:28.917030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.893872 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:28.946638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.941067 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:29.035934 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.981184 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:08:29.113795 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.112902 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:08:29.113795 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.982847 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:29.114119 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.085340 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:29.140666 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.085380 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:29.140666 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:28.982787 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 19:08:29.170386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.143332 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:08:29.170729 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.170690 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:08:29.178492 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.178278 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:08:29.178826 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.178793 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:08:29.188079 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.187945 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:29.189899 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.188668 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:08:29.191380 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.188722 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:29.193070 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.193050 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:08:29.959635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.947890 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:30.005636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.975909 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:08:30.005636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.992062 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:08:30.005636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:29.992125 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:08:30.054782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.054736 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:30.138296 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.138253 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:08:30.164060 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.164007 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:30.217578 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.174063 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 19:08:30.237783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.237734 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:08:30.285793 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.285741 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:30.340015 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.339975 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:08:30.342793 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.342747 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:08:30.472882 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.472784 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:08:30.482812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.481157 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:30.482812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.481252 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:30.482812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.481484 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:08:30.482812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.481518 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9"] Oct 30 19:08:30.482812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.482677 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" containerID="cri-o://7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581" gracePeriod=30 Oct 30 19:08:30.483748 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.483317 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:08:30.483748 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.483463 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:08:30.483947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.483919 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:08:30.484024 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.483972 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:08:30.484024 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.484003 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:08:30.484129 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.484031 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:08:30.484129 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.484094 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:08:30.484502 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.484402 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:30.484502 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.484467 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:08:30.484502 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.484485 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:30.484727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.484531 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:08:30.484727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.484572 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:08:30.848759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.848658 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:08:30.849031 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.848988 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:08:30.957560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.957515 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:08:30.957819 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:30.957574 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:08:31.838632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:31.838053 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:08:31.838632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:31.838118 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:08:31.895084 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:31.895049 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:08:31.895272 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:31.895103 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:08:32.844350 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:32.844316 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:32.844957 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:32.844859 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:32.845921 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:32.845892 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:08:32.903763 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:32.903731 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:32.904232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:32.904189 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:32.905217 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:32.905188 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:08:35.560451 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.560419 5959 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:08:35.560913 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.560489 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:08:35.761366 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.761338 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 19:08:35.805864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.805441 5959 generic.go:356] "Generic (PLEG): container finished" podID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerID="7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581" exitCode=0 Oct 30 19:08:35.805864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.805495 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" Oct 30 19:08:35.805864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.805545 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" event={"ID":"8c108933-c3fb-4fbd-97da-982149fedf1f","Type":"ContainerDied","Data":"7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581"} Oct 30 19:08:35.805864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.805660 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9" event={"ID":"8c108933-c3fb-4fbd-97da-982149fedf1f","Type":"ContainerDied","Data":"56dfeb9f3853074145606ee4de0f2aa4e9c0891268bbf8ba41d98c87f6f889ac"} Oct 30 19:08:35.805864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.805696 5959 scope.go:122] "RemoveContainer" containerID="7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581" Oct 30 19:08:35.823417 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.823391 5959 scope.go:122] "RemoveContainer" containerID="7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581" Oct 30 19:08:35.823871 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:08:35.823845 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581\": container with ID starting with 7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581 not found: ID does not exist" containerID="7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581" Oct 30 19:08:35.824008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.823881 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581"} err="failed to get container status \"7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581\": rpc error: code = NotFound desc = could not find container \"7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581\": container with ID starting with 7f8edd0c9393bd7a7ea3860fa6fb30e0fe77afc6c6c6aeb57b7c88041aac6581 not found: ID does not exist" Oct 30 19:08:35.857918 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.857882 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-envoy\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.858245 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858225 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-podinfo\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.858337 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858264 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-certs\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.858423 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858396 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-token\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-token\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.858506 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858487 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-kube-api-access-mfp44\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-kube-api-access-mfp44\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.858672 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858645 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-certs" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:35.858777 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858653 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c108933-c3fb-4fbd-97da-982149fedf1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c108933-c3fb-4fbd-97da-982149fedf1f-istiod-ca-cert\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.858777 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858734 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-data\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.858879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858779 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-socket\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.858978 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858949 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-data" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:35.859041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858991 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c108933-c3fb-4fbd-97da-982149fedf1f-istiod-ca-cert" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:35.859041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.858996 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-credential-socket\") pod \"8c108933-c3fb-4fbd-97da-982149fedf1f\" (UID: \"8c108933-c3fb-4fbd-97da-982149fedf1f\") " Oct 30 19:08:35.859265 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.859240 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-socket" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:35.859429 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.859291 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-credential-socket" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:35.859973 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.859947 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c108933-c3fb-4fbd-97da-982149fedf1f-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:35.860081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.859975 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:35.860081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.859989 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:35.860081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.860003 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:35.860081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.860016 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:35.860906 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.860887 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-envoy" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:35.861259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.861240 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-podinfo" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:35.861341 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.861316 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-kube-api-access-mfp44" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "kube-api-access-mfp44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:35.861691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.861669 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-token" pod "8c108933-c3fb-4fbd-97da-982149fedf1f" (UID: "8c108933-c3fb-4fbd-97da-982149fedf1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:35.961423 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.961390 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:35.961423 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.961418 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:35.961423 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.961427 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:35.961692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:35.961436 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfp44\" (UniqueName: \"kubernetes.io/projected/8c108933-c3fb-4fbd-97da-982149fedf1f-kube-api-access-mfp44\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:36.132856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.132823 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9"] Oct 30 19:08:36.136763 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.136732 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-qn4s9"] Oct 30 19:08:36.219672 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.219628 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l"] Oct 30 19:08:36.220816 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.220787 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" Oct 30 19:08:36.220816 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.220816 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" Oct 30 19:08:36.221061 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.221044 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" containerName="istio-proxy" Oct 30 19:08:36.227958 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.227916 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.232206 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.232173 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-28c1e9-serving-cert\"" Oct 30 19:08:36.238294 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.238202 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l"] Oct 30 19:08:36.266653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.266002 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-metrics-certs\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.266653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.266229 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lr9\" (UniqueName: \"kubernetes.io/projected/036f65e0-fe98-4601-b469-3c033588c81f-kube-api-access-t4lr9\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.266653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.266282 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/036f65e0-fe98-4601-b469-3c033588c81f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.266653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.266340 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.367623 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.367500 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-metrics-certs\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.367821 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.367705 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lr9\" (UniqueName: \"kubernetes.io/projected/036f65e0-fe98-4601-b469-3c033588c81f-kube-api-access-t4lr9\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.367821 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.367751 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/036f65e0-fe98-4601-b469-3c033588c81f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.367821 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.367785 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.368030 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:08:36.367885 5959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-serving-cert: secret "maas-api-e2e-extmod-b-28c1e9-serving-cert" not found Oct 30 19:08:36.368030 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:08:36.368000 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls podName:036f65e0-fe98-4601-b469-3c033588c81f nodeName:}" failed. No retries permitted until 2026-10-30 19:08:36.867975049 +0000 UTC m=+1897.196386433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls") pod "maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" (UID: "036f65e0-fe98-4601-b469-3c033588c81f") : secret "maas-api-e2e-extmod-b-28c1e9-serving-cert" not found Oct 30 19:08:36.368408 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.368388 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/036f65e0-fe98-4601-b469-3c033588c81f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.370638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.370592 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-metrics-certs\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.379227 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.379187 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lr9\" (UniqueName: \"kubernetes.io/projected/036f65e0-fe98-4601-b469-3c033588c81f-kube-api-access-t4lr9\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.883631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.883036 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:36.886890 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:36.886862 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:37.153783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:37.153742 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:37.470943 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:37.470908 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l"] Oct 30 19:08:37.471680 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:08:37.471654 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036f65e0_fe98_4601_b469_3c033588c81f.slice/crio-7facc0145e873d090efed1cc8d611dee9d4f486b789abed888b75d373d915840 WatchSource:0}: Error finding container 7facc0145e873d090efed1cc8d611dee9d4f486b789abed888b75d373d915840: Status 404 returned error can't find the container with id 7facc0145e873d090efed1cc8d611dee9d4f486b789abed888b75d373d915840 Oct 30 19:08:37.800508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:37.800423 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c108933-c3fb-4fbd-97da-982149fedf1f" path="/var/lib/kubelet/pods/8c108933-c3fb-4fbd-97da-982149fedf1f/volumes" Oct 30 19:08:37.900209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:37.900176 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" event={"ID":"036f65e0-fe98-4601-b469-3c033588c81f","Type":"ContainerStarted","Data":"7facc0145e873d090efed1cc8d611dee9d4f486b789abed888b75d373d915840"} Oct 30 19:08:38.912128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:38.912094 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" event={"ID":"036f65e0-fe98-4601-b469-3c033588c81f","Type":"ContainerStarted","Data":"b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab"} Oct 30 19:08:38.912557 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:38.912341 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:38.932500 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:38.932449 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" podStartSLOduration=2.507614219 podStartE2EDuration="2.932433662s" totalImagesPullingTime="424.819443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:38.929405409 +0000 UTC m=+1899.257816795" watchObservedRunningTime="2026-10-30 19:08:38.932433662 +0000 UTC m=+1899.260845078" Oct 30 19:08:40.134653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:40.134598 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:08:43.623241 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.623192 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz"] Oct 30 19:08:43.629773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.629727 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.641676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.636908 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e67d95\"/\"payload-processing-external-model-e2e--0b43b0f8-dockercfg-wv8pf\"" Oct 30 19:08:43.641676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.636991 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e67d95\"/\"openshift-service-ca.crt\"" Oct 30 19:08:43.641676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.637047 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e67d95\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e67d95\"" Oct 30 19:08:43.641676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.637109 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e67d95\"/\"kube-root-ca.crt\"" Oct 30 19:08:43.641676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.637184 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e67d95\"/\"routing-overlay\"" Oct 30 19:08:43.641676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.637239 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e67d95\"/\"agc-cred-191a998f1cf1\"" Oct 30 19:08:43.641676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.640784 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz"] Oct 30 19:08:43.736872 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.736829 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.737211 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.737187 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.737269 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.737229 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.737269 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.737262 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbpbc\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-kube-api-access-wbpbc\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.839482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.839436 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.839758 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.839503 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.839758 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.839552 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbpbc\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-kube-api-access-wbpbc\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.839758 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.839694 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.840650 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.840622 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.841258 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.841230 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.871701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.868462 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbpbc\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-kube-api-access-wbpbc\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.881688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.872569 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:43.989696 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:43.959026 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:44.452017 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:44.451989 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz"] Oct 30 19:08:44.454097 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:08:44.454062 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76b9e4b_12b0_438f_8952_4619872de7d1.slice/crio-aacb6e3f615265feba28fdf3818e53fd83d49617c389862c53a18d1a3cc82285 WatchSource:0}: Error finding container aacb6e3f615265feba28fdf3818e53fd83d49617c389862c53a18d1a3cc82285: Status 404 returned error can't find the container with id aacb6e3f615265feba28fdf3818e53fd83d49617c389862c53a18d1a3cc82285 Oct 30 19:08:44.925764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:44.925730 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:08:45.035700 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:45.035662 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" event={"ID":"f76b9e4b-12b0-438f-8952-4619872de7d1","Type":"ContainerStarted","Data":"c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e"} Oct 30 19:08:45.035862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:45.035719 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" event={"ID":"f76b9e4b-12b0-438f-8952-4619872de7d1","Type":"ContainerStarted","Data":"aacb6e3f615265feba28fdf3818e53fd83d49617c389862c53a18d1a3cc82285"} Oct 30 19:08:45.035862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:45.035738 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:45.056093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:45.056031 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" podStartSLOduration=2.056013786 podStartE2EDuration="2.056013786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:45.052074066 +0000 UTC m=+1905.380485452" watchObservedRunningTime="2026-10-30 19:08:45.056013786 +0000 UTC m=+1905.384425195" Oct 30 19:08:55.331039 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.331001 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq"] Oct 30 19:08:55.341661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.341580 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.344487 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.344452 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq"] Oct 30 19:08:55.349477 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.349446 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c164d6\"/\"openshift-service-ca.crt\"" Oct 30 19:08:55.349733 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.349568 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c164d6\"/\"kube-root-ca.crt\"" Oct 30 19:08:55.349854 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.349590 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c164d6\"/\"autores-model-c164d6-kserve-self-signed-certs\"" Oct 30 19:08:55.349907 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.349689 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c164d6\"/\"default-dockercfg-hxxlz\"" Oct 30 19:08:55.431200 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.431164 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-model-cache\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.431372 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.431218 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-home\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.431372 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.431319 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-tmp-dir\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.431372 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.431365 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2mv\" (UniqueName: \"kubernetes.io/projected/7d881827-4f39-4fa5-967c-3d725e98e4f4-kube-api-access-bz2mv\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.431533 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.431518 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d881827-4f39-4fa5-967c-3d725e98e4f4-tls-certs\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.431663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.431646 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-dshm\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.546628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.545202 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d881827-4f39-4fa5-967c-3d725e98e4f4-tls-certs\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.546628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.545910 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-dshm\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.546628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.545996 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-model-cache\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.546628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.546055 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-home\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.546628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.546106 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-tmp-dir\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.546628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.546138 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2mv\" (UniqueName: \"kubernetes.io/projected/7d881827-4f39-4fa5-967c-3d725e98e4f4-kube-api-access-bz2mv\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.547096 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.547053 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-model-cache\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.550668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.547964 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-home\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.550668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.548206 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-tmp-dir\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.579866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.579835 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-dshm\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.582760 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.582694 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2mv\" (UniqueName: \"kubernetes.io/projected/7d881827-4f39-4fa5-967c-3d725e98e4f4-kube-api-access-bz2mv\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.583119 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.583099 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d881827-4f39-4fa5-967c-3d725e98e4f4-tls-certs\") pod \"autores-model-c164d6-kserve-77757c757c-fr6tq\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.666106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.663524 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:55.983193 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:55.983152 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq"] Oct 30 19:08:55.985655 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:08:55.985625 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d881827_4f39_4fa5_967c_3d725e98e4f4.slice/crio-8c2ea864ec9578191792176b5ebdada528e5e995d848e740121d00d001f86f24 WatchSource:0}: Error finding container 8c2ea864ec9578191792176b5ebdada528e5e995d848e740121d00d001f86f24: Status 404 returned error can't find the container with id 8c2ea864ec9578191792176b5ebdada528e5e995d848e740121d00d001f86f24 Oct 30 19:08:56.047053 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:56.047022 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:08:56.145183 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:56.145144 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" event={"ID":"7d881827-4f39-4fa5-967c-3d725e98e4f4","Type":"ContainerStarted","Data":"bbedc591ba1ba6a6d3ac91deaf71de03aeb1e428c71ea8d0164bf59ef9495eb6"} Oct 30 19:08:56.145439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:56.145418 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" event={"ID":"7d881827-4f39-4fa5-967c-3d725e98e4f4","Type":"ContainerStarted","Data":"8c2ea864ec9578191792176b5ebdada528e5e995d848e740121d00d001f86f24"} Oct 30 19:08:56.166284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:56.166224 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" podStartSLOduration=1.166205925 podStartE2EDuration="1.166205925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:56.162059725 +0000 UTC m=+1916.490471132" watchObservedRunningTime="2026-10-30 19:08:56.166205925 +0000 UTC m=+1916.494617306" Oct 30 19:08:56.666589 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:08:56.666548 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:08:57.816898 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:08:57.816860 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:09:01.234297 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:01.234259 5959 scope.go:122] "RemoveContainer" containerID="f566bc1d57c95bfac8ad96a95230550b27112ba690470168397aab5f4c5cd258" Oct 30 19:09:03.339656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.339596 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s"] Oct 30 19:09:03.346560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.346530 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.356790 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.356476 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28c1e9\"/\"agc-cred-e01a294f7001\"" Oct 30 19:09:03.357166 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.357144 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28c1e9\"/\"openshift-service-ca.crt\"" Oct 30 19:09:03.357595 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.357534 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28c1e9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-28c1e9\"" Oct 30 19:09:03.357595 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.357586 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28c1e9\"/\"kube-root-ca.crt\"" Oct 30 19:09:03.357793 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.357585 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28c1e9\"/\"routing-overlay\"" Oct 30 19:09:03.363349 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.363320 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s"] Oct 30 19:09:03.402812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.402776 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.403031 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.402885 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.403031 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.402909 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.403116 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.403082 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njd5\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-kube-api-access-9njd5\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.516480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.511260 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.516480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.511350 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.516480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.511384 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.516480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.511555 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njd5\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-kube-api-access-9njd5\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.516480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.514506 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.534629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.530028 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.534629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.532409 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.564631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.564350 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njd5\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-kube-api-access-9njd5\") pod \"payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:03.672633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:03.670208 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:04.026446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:04.026415 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s"] Oct 30 19:09:04.028143 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:09:04.028114 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e94bd6_1134_498a_b3d9_fab0acb86329.slice/crio-2c5583f0e7c6ddcf1f9dfa057b266d073950a199092862cb2658d4b2d0760a04 WatchSource:0}: Error finding container 2c5583f0e7c6ddcf1f9dfa057b266d073950a199092862cb2658d4b2d0760a04: Status 404 returned error can't find the container with id 2c5583f0e7c6ddcf1f9dfa057b266d073950a199092862cb2658d4b2d0760a04 Oct 30 19:09:04.209988 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:04.209954 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" event={"ID":"50e94bd6-1134-498a-b3d9-fab0acb86329","Type":"ContainerStarted","Data":"5ff6f60a21e8056bde92a6f144500b2be5153061da4b55293b8c78c75a3950c2"} Oct 30 19:09:04.210317 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:04.210020 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" event={"ID":"50e94bd6-1134-498a-b3d9-fab0acb86329","Type":"ContainerStarted","Data":"2c5583f0e7c6ddcf1f9dfa057b266d073950a199092862cb2658d4b2d0760a04"} Oct 30 19:09:04.210317 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:04.210053 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:04.240188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:04.240121 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" podStartSLOduration=1.240097515 podStartE2EDuration="1.240097515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:04.234238997 +0000 UTC m=+1924.562650383" watchObservedRunningTime="2026-10-30 19:09:04.240097515 +0000 UTC m=+1924.568508899" Oct 30 19:09:05.665822 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:05.665788 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:09:05.679502 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:05.679473 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:09:06.238835 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:06.238804 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:09:08.677105 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.677067 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv"] Oct 30 19:09:08.683107 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.683034 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.704566 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.704528 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv"] Oct 30 19:09:08.802141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802064 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.802340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802167 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.802340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802210 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.802441 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802331 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.802441 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802359 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e75b5f-85d1-4d91-8b25-2890951b57f8-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.802511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802475 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7sm\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-kube-api-access-gt7sm\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.802511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802498 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.802594 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802526 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.802691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.802643 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.904648 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.904560 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.904866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.904704 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.904866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.904768 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.904866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.904845 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.904865 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e75b5f-85d1-4d91-8b25-2890951b57f8-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.904994 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7sm\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-kube-api-access-gt7sm\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.905031 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.905070 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.905068 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.905147 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.905192 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905415 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.905372 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905514 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.905488 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.905849 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.905822 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e75b5f-85d1-4d91-8b25-2890951b57f8-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.907570 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.907549 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.907713 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.907696 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.913913 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.913891 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:08.914134 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:08.914097 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7sm\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-kube-api-access-gt7sm\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:09.014028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:09.013947 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:09.309342 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:09.309311 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv"] Oct 30 19:09:09.310458 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:09:09.310403 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e75b5f_85d1_4d91_8b25_2890951b57f8.slice/crio-c69c8e498cd139b76e8419abc2a8857fe15b514cced3a04d511f49d9549cd153 WatchSource:0}: Error finding container c69c8e498cd139b76e8419abc2a8857fe15b514cced3a04d511f49d9549cd153: Status 404 returned error can't find the container with id c69c8e498cd139b76e8419abc2a8857fe15b514cced3a04d511f49d9549cd153 Oct 30 19:09:09.314174 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:09.314129 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:09.314288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:09.314236 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:09.314328 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:09.314315 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:10.004718 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:10.003646 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq"] Oct 30 19:09:10.269665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:10.269548 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" event={"ID":"61e75b5f-85d1-4d91-8b25-2890951b57f8","Type":"ContainerStarted","Data":"ae38a8ef0b2fcbbe5d6a42c951dcceee95401ca8e6339b5ba4f425d8cbc601fc"} Oct 30 19:09:10.269665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:10.269635 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" event={"ID":"61e75b5f-85d1-4d91-8b25-2890951b57f8","Type":"ContainerStarted","Data":"c69c8e498cd139b76e8419abc2a8857fe15b514cced3a04d511f49d9549cd153"} Oct 30 19:09:10.312578 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:10.312511 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" podStartSLOduration=2.312488312 podStartE2EDuration="2.312488312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:10.308081165 +0000 UTC m=+1930.636492551" watchObservedRunningTime="2026-10-30 19:09:10.312488312 +0000 UTC m=+1930.640899698" Oct 30 19:09:11.021639 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:11.016060 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:11.021639 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:11.020777 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:09:11.021639 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:11.020844 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:09:12.016138 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:12.016093 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:09:12.016458 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:12.016425 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:09:13.017632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.017078 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:09:13.017632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.017164 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:09:13.139303 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.139264 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf"] Oct 30 19:09:13.145622 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.145582 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.155156 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.155120 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c164d6-openshift-default-dockercfg-c42sr\"" Oct 30 19:09:13.157121 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.157094 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf"] Oct 30 19:09:13.163142 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.163073 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc"] Oct 30 19:09:13.171809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.170029 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193007 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.192967 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.193188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193032 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsm5d\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-kube-api-access-jsm5d\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.193188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193137 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193168 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193365 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193195 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.193365 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193234 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.193365 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193299 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.193520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193400 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.193520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193472 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193505 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193545 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.193704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193595 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed381eeb-43ce-4084-9ee7-1436cee51197-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.193704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193641 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193669 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193912 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193757 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193912 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193807 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzw75\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-kube-api-access-mzw75\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193912 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193870 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.193912 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.193898 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.197126 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.197096 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc"] Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296035 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296096 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296147 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsm5d\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-kube-api-access-jsm5d\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296284 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296323 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296355 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296397 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296452 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296542 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296636 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296669 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296712 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296763 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed381eeb-43ce-4084-9ee7-1436cee51197-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296792 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296820 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296916 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.296968 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzw75\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-kube-api-access-mzw75\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.297036 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.299684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.298180 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.317956 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.318027 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.318253 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.318310 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.318597 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.318944 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed381eeb-43ce-4084-9ee7-1436cee51197-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.319190 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.319199 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.323686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.320549 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.328273 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.328238 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.331205 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.330622 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.331750 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.331723 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzw75\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-kube-api-access-mzw75\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.332162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.332133 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.333052 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.333029 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.333262 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.333240 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.333371 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.333349 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.335546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.335514 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsm5d\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-kube-api-access-jsm5d\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.471071 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.471033 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:13.526635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.523700 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:13.844640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.844515 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf"] Oct 30 19:09:13.846141 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:09:13.846093 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded381eeb_43ce_4084_9ee7_1436cee51197.slice/crio-d9709787200fa581caeea4ef32aa878a2d319adbf18e1f3c97d13bafac901dd6 WatchSource:0}: Error finding container d9709787200fa581caeea4ef32aa878a2d319adbf18e1f3c97d13bafac901dd6: Status 404 returned error can't find the container with id d9709787200fa581caeea4ef32aa878a2d319adbf18e1f3c97d13bafac901dd6 Oct 30 19:09:13.851994 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.851952 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:13.852140 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.852043 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:13.852461 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.852432 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:13.878131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.877860 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc"] Oct 30 19:09:13.879210 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:09:13.879182 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fce9c0_114c_4aae_8ccf_33720a9cd1e3.slice/crio-af2f24d1f64c2e1ba50a46d221223cd127f03e756c31dcb60f619f0cf3075468 WatchSource:0}: Error finding container af2f24d1f64c2e1ba50a46d221223cd127f03e756c31dcb60f619f0cf3075468: Status 404 returned error can't find the container with id af2f24d1f64c2e1ba50a46d221223cd127f03e756c31dcb60f619f0cf3075468 Oct 30 19:09:13.882587 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.882556 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:13.882700 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.882669 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:13.882743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:13.882709 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:14.028225 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.028173 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:09:14.028989 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.028926 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:09:14.351786 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.351743 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" event={"ID":"ed381eeb-43ce-4084-9ee7-1436cee51197","Type":"ContainerStarted","Data":"5bd0787b60598f34e7ce78f75eff6e8b38c68fdcded69a1d6fbd2c33af80b2cf"} Oct 30 19:09:14.352072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.352047 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" event={"ID":"ed381eeb-43ce-4084-9ee7-1436cee51197","Type":"ContainerStarted","Data":"d9709787200fa581caeea4ef32aa878a2d319adbf18e1f3c97d13bafac901dd6"} Oct 30 19:09:14.355771 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.355732 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" event={"ID":"69fce9c0-114c-4aae-8ccf-33720a9cd1e3","Type":"ContainerStarted","Data":"314bb93864e8cfdf1d5dc9bd0cda27245c036cdd22b0630ac1d2d2dee49f1fd2"} Oct 30 19:09:14.355965 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.355799 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" event={"ID":"69fce9c0-114c-4aae-8ccf-33720a9cd1e3","Type":"ContainerStarted","Data":"af2f24d1f64c2e1ba50a46d221223cd127f03e756c31dcb60f619f0cf3075468"} Oct 30 19:09:14.450686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.450550 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podStartSLOduration=1.4505293799999999 podStartE2EDuration="1.45052938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:14.385373439 +0000 UTC m=+1934.713784825" watchObservedRunningTime="2026-10-30 19:09:14.45052938 +0000 UTC m=+1934.778940768" Oct 30 19:09:14.461008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.460924 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podStartSLOduration=1.460900783 podStartE2EDuration="1.460900783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:14.442529274 +0000 UTC m=+1934.770940658" watchObservedRunningTime="2026-10-30 19:09:14.460900783 +0000 UTC m=+1934.789312171" Oct 30 19:09:14.471568 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.471532 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:14.473243 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.473175 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 19:09:14.473408 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.473254 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 19:09:14.529248 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:14.525750 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:15.024108 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.024065 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:09:15.024394 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.024359 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:09:15.255631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.247831 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:09:15.536679 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.519535 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 19:09:15.536679 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.519595 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 19:09:15.647630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.587870 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:09:15.648002 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.647948 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:09:15.695372 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.695335 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5"] Oct 30 19:09:15.873110 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.873010 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:15.881012 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.880958 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e67d95-openshift-default-dockercfg-jdrgt\"" Oct 30 19:09:15.891371 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.891339 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh"] Oct 30 19:09:15.925631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.920746 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5"] Oct 30 19:09:15.925631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.920805 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh"] Oct 30 19:09:15.925631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:15.920919 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.030537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.030494 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75s8w\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-kube-api-access-75s8w\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.035631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.031300 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a43000-522d-4ea3-9d38-4ceaa5789aea-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.035940 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.045793 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.045950 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046020 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359be520-f69e-4d59-8a67-55e6a11236a5-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046123 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046269 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046316 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046351 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046429 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046477 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67s7\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-kube-api-access-p67s7\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046535 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046667 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046711 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046804 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359be520-f69e-4d59-8a67-55e6a11236a5-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046825 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.046939 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.030927 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:09:16.052221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.047343 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.155897 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359be520-f69e-4d59-8a67-55e6a11236a5-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.155955 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156041 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156132 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75s8w\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-kube-api-access-75s8w\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156168 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a43000-522d-4ea3-9d38-4ceaa5789aea-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156204 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156232 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156312 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156355 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359be520-f69e-4d59-8a67-55e6a11236a5-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156431 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156537 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156568 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156597 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156676 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156706 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67s7\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-kube-api-access-p67s7\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156747 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156819 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.156853 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.158159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.157492 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.159416 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.158287 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359be520-f69e-4d59-8a67-55e6a11236a5-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.160713 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.158629 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.160909 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.159290 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.161031 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.160393 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a43000-522d-4ea3-9d38-4ceaa5789aea-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.161141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.160726 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.161420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.161394 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.161780 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.161751 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.162444 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.162007 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.162673 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.162651 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.164918 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.164889 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.165285 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.165259 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359be520-f69e-4d59-8a67-55e6a11236a5-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.166044 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.166020 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.166402 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.166355 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.172315 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.172289 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.173056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.173033 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67s7\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-kube-api-access-p67s7\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.173552 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.173528 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75s8w\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-kube-api-access-75s8w\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.173907 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.173885 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.249787 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.245072 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:16.290979 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.290931 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:16.473137 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.472649 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 19:09:16.473137 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.472707 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 19:09:16.546406 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.545912 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:09:16.546406 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.545981 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:09:16.965494 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:09:16.965447 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359be520_f69e_4d59_8a67_55e6a11236a5.slice/crio-53a544ade5c9330720e9e33dbf4cd1d1538bb8035916e9cbc719a72621a1a549 WatchSource:0}: Error finding container 53a544ade5c9330720e9e33dbf4cd1d1538bb8035916e9cbc719a72621a1a549: Status 404 returned error can't find the container with id 53a544ade5c9330720e9e33dbf4cd1d1538bb8035916e9cbc719a72621a1a549 Oct 30 19:09:16.965716 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.965540 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5"] Oct 30 19:09:16.969473 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.969440 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:16.969637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.969512 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:16.969637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:16.969547 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:17.053657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.050515 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:17.055277 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.054217 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:17.057915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.055738 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:09:17.193010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.192942 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:17.193225 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.193121 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:17.193225 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.193211 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:17.273706 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.238573 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh"] Oct 30 19:09:17.471526 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.471483 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 19:09:17.471933 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.471552 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 19:09:17.530281 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.530244 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:09:17.530907 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.530505 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:09:17.746382 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.746280 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" event={"ID":"48a43000-522d-4ea3-9d38-4ceaa5789aea","Type":"ContainerStarted","Data":"1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1"} Oct 30 19:09:17.756743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.756688 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" event={"ID":"48a43000-522d-4ea3-9d38-4ceaa5789aea","Type":"ContainerStarted","Data":"b1e4f57379516917889e796fbad4b3170f6065b368daaa782eacdf49ca59eb44"} Oct 30 19:09:17.787344 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.787224 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" event={"ID":"359be520-f69e-4d59-8a67-55e6a11236a5","Type":"ContainerStarted","Data":"6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3"} Oct 30 19:09:17.787344 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.787311 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" event={"ID":"359be520-f69e-4d59-8a67-55e6a11236a5","Type":"ContainerStarted","Data":"53a544ade5c9330720e9e33dbf4cd1d1538bb8035916e9cbc719a72621a1a549"} Oct 30 19:09:17.875703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.802865 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podStartSLOduration=2.802843891 podStartE2EDuration="2.802843891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:17.801081567 +0000 UTC m=+1938.129492953" watchObservedRunningTime="2026-10-30 19:09:17.802843891 +0000 UTC m=+1938.131255278" Oct 30 19:09:17.895304 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:17.894878 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podStartSLOduration=2.894857532 podStartE2EDuration="2.894857532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:17.891003523 +0000 UTC m=+1938.219414904" watchObservedRunningTime="2026-10-30 19:09:17.894857532 +0000 UTC m=+1938.223268919" Oct 30 19:09:18.248634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.248400 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:18.248634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.248547 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 19:09:18.248634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.248599 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 19:09:18.292721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.292687 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:18.293911 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.293880 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:18.294056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.293936 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:18.472075 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.472033 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 19:09:18.472450 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.472102 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 19:09:18.525565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.525048 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:09:18.525565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:18.525118 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:09:19.251632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:19.251062 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 19:09:19.251632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:19.251129 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 19:09:19.347794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:19.347538 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:19.348106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:19.348075 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:19.516692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:19.475865 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 19:09:19.523086 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:19.523040 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 19:09:19.579448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:19.558432 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:09:19.579790 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:19.579753 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:09:20.144634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.120317 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:20.144634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.120926 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 19:09:20.165646 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.145567 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:20.290632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.264835 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.138:8000/ready\": context deadline exceeded" Oct 30 19:09:20.320937 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.318745 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:20.321226 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.321197 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:20.321323 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.318792 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 19:09:20.321407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.321393 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 19:09:20.406922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.394787 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:20.437539 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.437503 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:09:20.492791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.492755 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 19:09:20.524150 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.524104 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 19:09:20.590196 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.522099 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:20.622929 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.572380 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:09:20.624130 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.624092 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:09:20.913995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.913941 5959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:20.940938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:20.930403 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:21.036640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.033889 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:21.036974 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.036930 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:21.071654 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.062715 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:21.127482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.127441 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 19:09:21.128841 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.128817 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:09:21.184301 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.184176 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:09:21.185066 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.185041 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:09:21.210662 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.200709 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:21.210662 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.201097 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:09:21.283668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.283620 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 19:09:21.284222 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.284019 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 19:09:21.301184 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.298859 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:21.304632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.301470 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:21.335291 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.335187 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.138:8000/ready\": context deadline exceeded" Oct 30 19:09:21.336022 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.335994 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:09:21.482318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.482215 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 19:09:21.482590 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.482555 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 19:09:21.524380 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.524342 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:09:21.776409 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.764646 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:09:21.828158 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.714082 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:21.972029 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:21.971975 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 19:09:22.082301 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.082192 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:22.185084 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.185037 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:22.194944 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.194900 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:09:22.232771 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.232715 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:22.356438 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.356345 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 19:09:22.356749 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.356717 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 19:09:22.366688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.366642 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.138:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:22.383623 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.383566 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:22.400635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.396554 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:22.404661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.400902 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:22.539658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.531526 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:09:22.594685 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.594652 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:09:22.604185 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.604156 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:09:22.604702 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.604681 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7"] Oct 30 19:09:22.786742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.765449 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:09:22.786742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.765494 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:09:22.786742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.765526 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:09:22.786742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.765559 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:09:22.786742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.765593 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:09:22.786742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.766021 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.786742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.771750 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7"] Oct 30 19:09:22.887638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.887012 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:09:22.916371 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.909673 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.917233 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.917185 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rpw7\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-kube-api-access-2rpw7\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.917968 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.917842 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.918477 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.918459 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.920723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.920686 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.921328 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.921305 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.921554 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.921537 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03100ce6-ad8c-4e96-8403-1da2674f46b5-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.921802 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.921785 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.921976 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.921961 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:22.948094 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:22.948064 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072038 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072115 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03100ce6-ad8c-4e96-8403-1da2674f46b5-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072160 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072189 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072228 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072276 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rpw7\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-kube-api-access-2rpw7\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072354 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072431 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.072658 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.109421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.108922 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.126295 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.118850 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03100ce6-ad8c-4e96-8403-1da2674f46b5-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.148093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.126961 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.148093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.128787 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.170991 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.170949 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.254094 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.215319 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.254094 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.218579 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rpw7\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-kube-api-access-2rpw7\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.273025 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.269622 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 19:09:23.273025 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.270337 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:09:23.286637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.284755 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:23.297331 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.297295 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 19:09:23.297531 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.297374 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 19:09:23.365681 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.329908 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:23.395640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.393694 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:23.490735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.490691 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:23.614709 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.614672 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:09:23.652260 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.652219 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:23.689907 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.689866 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:09:23.690451 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.689954 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:23.708890 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.692735 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:09:23.968689 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:23.965880 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:24.023306 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.023225 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:24.079573 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.040378 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:24.094012 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.082319 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:09:24.140483 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.140415 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:24.157395 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.156778 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:24.163172 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.163137 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:24.191432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.191393 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:09:24.200642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.197091 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:24.200642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.199322 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:24.251688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.247943 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:24.253679 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.251876 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.138:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:09:24.274645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.271839 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 19:09:24.274645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.271908 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 19:09:24.340435 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.326956 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:24.340435 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.327295 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:24.574818 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.574781 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72"] Oct 30 19:09:24.584707 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.583725 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.586625 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.586580 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72"] Oct 30 19:09:24.786805 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.786756 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.787456 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.787421 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.787835 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.787792 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.788001 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.787983 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.788216 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.788201 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftw6q\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-kube-api-access-ftw6q\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.788387 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.788371 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.788485 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.788472 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.788780 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.788763 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.788922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.788907 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.890883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.890849 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.891096 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.890933 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.891096 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.891037 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.891096 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.891059 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.891275 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.891159 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftw6q\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-kube-api-access-ftw6q\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.891275 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.891251 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.891387 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.891294 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.891684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.891657 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.891954 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.891917 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.892151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.892131 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.892534 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.892479 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.892698 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.892681 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.892774 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.892711 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.893348 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.893320 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.895425 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.895400 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.941284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.941216 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.973546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.959362 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftw6q\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-kube-api-access-ftw6q\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.974321 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.974061 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:24.992857 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:09:24.992823 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03100ce6_ad8c_4e96_8403_1da2674f46b5.slice/crio-392b92f9f0e98c637152b193bbd6cd42c5f0300504b40d788ae96b267bbc3b7d WatchSource:0}: Error finding container 392b92f9f0e98c637152b193bbd6cd42c5f0300504b40d788ae96b267bbc3b7d: Status 404 returned error can't find the container with id 392b92f9f0e98c637152b193bbd6cd42c5f0300504b40d788ae96b267bbc3b7d Oct 30 19:09:24.993172 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:24.993144 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7"] Oct 30 19:09:25.000219 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.000164 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:25.000521 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.000495 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:25.031798 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.031748 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:25.051917 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.051841 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" containerID="cri-o://bbedc591ba1ba6a6d3ac91deaf71de03aeb1e428c71ea8d0164bf59ef9495eb6" gracePeriod=45 Oct 30 19:09:25.094916 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.094883 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:25.237338 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.237289 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.138:8000/ready\": dial tcp 10.134.0.138:8000: connect: connection refused" Oct 30 19:09:25.259367 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.246465 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 19:09:25.259632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.259576 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 19:09:25.292231 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.292201 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:25.292302 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.292259 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:25.633051 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.631466 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" event={"ID":"03100ce6-ad8c-4e96-8403-1da2674f46b5","Type":"ContainerStarted","Data":"53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35"} Oct 30 19:09:25.633051 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.631542 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" event={"ID":"03100ce6-ad8c-4e96-8403-1da2674f46b5","Type":"ContainerStarted","Data":"392b92f9f0e98c637152b193bbd6cd42c5f0300504b40d788ae96b267bbc3b7d"} Oct 30 19:09:25.636300 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.636218 5959 generic.go:356] "Generic (PLEG): container finished" podID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerID="bbedc591ba1ba6a6d3ac91deaf71de03aeb1e428c71ea8d0164bf59ef9495eb6" exitCode=0 Oct 30 19:09:25.647652 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.640689 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" event={"ID":"7d881827-4f39-4fa5-967c-3d725e98e4f4","Type":"ContainerDied","Data":"bbedc591ba1ba6a6d3ac91deaf71de03aeb1e428c71ea8d0164bf59ef9495eb6"} Oct 30 19:09:25.763633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.761345 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podStartSLOduration=4.761320581 podStartE2EDuration="4.761320581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:25.726824374 +0000 UTC m=+1946.055235762" watchObservedRunningTime="2026-10-30 19:09:25.761320581 +0000 UTC m=+1946.089731967" Oct 30 19:09:25.783334 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.783294 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72"] Oct 30 19:09:25.788557 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.788494 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:25.789209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.788732 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:25.789209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:25.788875 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:26.060207 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.059059 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:26.060817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.060064 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:09:26.060817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.060541 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:09:26.143707 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.118427 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:09:26.249703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.247089 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d881827-4f39-4fa5-967c-3d725e98e4f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d881827-4f39-4fa5-967c-3d725e98e4f4-tls-certs\") pod \"7d881827-4f39-4fa5-967c-3d725e98e4f4\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " Oct 30 19:09:26.249703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.247190 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-tmp-dir\") pod \"7d881827-4f39-4fa5-967c-3d725e98e4f4\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " Oct 30 19:09:26.249703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.247460 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-model-cache\") pod \"7d881827-4f39-4fa5-967c-3d725e98e4f4\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " Oct 30 19:09:26.249703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.247495 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-home\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-home\") pod \"7d881827-4f39-4fa5-967c-3d725e98e4f4\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " Oct 30 19:09:26.249703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.247637 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d881827-4f39-4fa5-967c-3d725e98e4f4-kube-api-access-bz2mv\" (UniqueName: \"kubernetes.io/projected/7d881827-4f39-4fa5-967c-3d725e98e4f4-kube-api-access-bz2mv\") pod \"7d881827-4f39-4fa5-967c-3d725e98e4f4\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " Oct 30 19:09:26.249703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.247700 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-dshm\") pod \"7d881827-4f39-4fa5-967c-3d725e98e4f4\" (UID: \"7d881827-4f39-4fa5-967c-3d725e98e4f4\") " Oct 30 19:09:26.249703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.249526 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-model-cache" pod "7d881827-4f39-4fa5-967c-3d725e98e4f4" (UID: "7d881827-4f39-4fa5-967c-3d725e98e4f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.250417 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.250257 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-home" pod "7d881827-4f39-4fa5-967c-3d725e98e4f4" (UID: "7d881827-4f39-4fa5-967c-3d725e98e4f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.250901 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.250597 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-tmp-dir" pod "7d881827-4f39-4fa5-967c-3d725e98e4f4" (UID: "7d881827-4f39-4fa5-967c-3d725e98e4f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.285823 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.285765 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-dshm" pod "7d881827-4f39-4fa5-967c-3d725e98e4f4" (UID: "7d881827-4f39-4fa5-967c-3d725e98e4f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.286733 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.286685 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d881827-4f39-4fa5-967c-3d725e98e4f4-kube-api-access-bz2mv" pod "7d881827-4f39-4fa5-967c-3d725e98e4f4" (UID: "7d881827-4f39-4fa5-967c-3d725e98e4f4"). InnerVolumeSpecName "kube-api-access-bz2mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:26.290446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.290325 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d881827-4f39-4fa5-967c-3d725e98e4f4-tls-certs" pod "7d881827-4f39-4fa5-967c-3d725e98e4f4" (UID: "7d881827-4f39-4fa5-967c-3d725e98e4f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:26.292443 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.292415 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:26.292703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.292673 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:26.350813 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.350775 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:26.350813 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.350811 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:26.351040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.350824 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2mv\" (UniqueName: \"kubernetes.io/projected/7d881827-4f39-4fa5-967c-3d725e98e4f4-kube-api-access-bz2mv\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:26.351040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.350837 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:26.351040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.350848 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d881827-4f39-4fa5-967c-3d725e98e4f4-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:26.351040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.350861 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d881827-4f39-4fa5-967c-3d725e98e4f4-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:26.681634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.681553 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" event={"ID":"81fa02a0-22e8-4b53-924b-19e6cb605d5b","Type":"ContainerStarted","Data":"2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb"} Oct 30 19:09:26.681854 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.681693 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" event={"ID":"81fa02a0-22e8-4b53-924b-19e6cb605d5b","Type":"ContainerStarted","Data":"5e20afe1d01572ca35783b693b860ae2bebf4a9a9b04eea4cf2633eff46f2a18"} Oct 30 19:09:26.701635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.685567 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" Oct 30 19:09:26.701635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.685779 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq" event={"ID":"7d881827-4f39-4fa5-967c-3d725e98e4f4","Type":"ContainerDied","Data":"8c2ea864ec9578191792176b5ebdada528e5e995d848e740121d00d001f86f24"} Oct 30 19:09:26.701635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.686006 5959 scope.go:122] "RemoveContainer" containerID="bbedc591ba1ba6a6d3ac91deaf71de03aeb1e428c71ea8d0164bf59ef9495eb6" Oct 30 19:09:26.750171 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.750104 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podStartSLOduration=3.7500813429999997 podStartE2EDuration="3.750081343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:26.708457858 +0000 UTC m=+1947.036869244" watchObservedRunningTime="2026-10-30 19:09:26.750081343 +0000 UTC m=+1947.078492729" Oct 30 19:09:26.817632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.815692 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq"] Oct 30 19:09:26.824391 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.818848 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c164d6/autores-model-c164d6-kserve-77757c757c-fr6tq"] Oct 30 19:09:26.851653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.848970 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:26.851994 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.851970 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:26.853386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:26.853350 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:09:27.058388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.058292 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:09:27.058388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.058349 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:09:27.120670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.115681 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:27.120670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.117340 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:09:27.120670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.117400 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:09:27.323688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.322970 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:27.326357 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.324302 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:27.331708 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.329935 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:09:27.870442 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:09:27.870168 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:09:27.873261 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:27.872686 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" path="/var/lib/kubelet/pods/7d881827-4f39-4fa5-967c-3d725e98e4f4/volumes" Oct 30 19:09:28.061893 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:28.060542 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:09:28.061893 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:28.060648 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:09:28.130689 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:28.129767 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:09:28.130689 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:28.129837 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:09:29.062996 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:29.061765 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:09:29.062996 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:29.061855 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:09:29.095329 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:29.095295 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:09:29.095559 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:29.095345 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:09:30.058244 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:30.058210 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:09:30.058437 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:30.058264 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:09:30.095678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:30.095574 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:09:30.116745 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:30.116689 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:09:30.683246 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:30.683205 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs"] Oct 30 19:09:31.098690 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:31.057859 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:09:31.259980 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:31.259932 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:09:31.260266 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:31.233370 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:09:31.260414 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:31.260391 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:09:32.058238 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:32.058200 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:09:32.058429 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:32.058253 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:09:32.103633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:32.095570 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:09:32.103633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:32.095659 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:09:33.063389 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:33.063355 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:33.063949 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:33.063919 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:33.065309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:33.065175 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:09:33.122228 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:33.122175 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:09:33.122778 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:33.122252 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:09:34.100908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:34.100875 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:34.101390 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:34.101368 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:34.102250 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:34.102227 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:09:35.686785 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:35.686738 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" podUID="3d484e54-c577-47a7-a127-33548ae935bf" containerName="payload-pre-processing" containerID="cri-o://3c2b48e7b36412a348a762dfda3762f9b918376d890b0073d89d68e7dbbc597a" gracePeriod=25 Oct 30 19:09:35.910270 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:35.908900 5959 generic.go:356] "Generic (PLEG): container finished" podID="3d484e54-c577-47a7-a127-33548ae935bf" containerID="3c2b48e7b36412a348a762dfda3762f9b918376d890b0073d89d68e7dbbc597a" exitCode=0 Oct 30 19:09:35.910270 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:35.909255 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" event={"ID":"3d484e54-c577-47a7-a127-33548ae935bf","Type":"ContainerDied","Data":"3c2b48e7b36412a348a762dfda3762f9b918376d890b0073d89d68e7dbbc597a"} Oct 30 19:09:36.008285 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.008254 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:09:36.056688 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.056651 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n"] Oct 30 19:09:36.059091 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.058909 5959 cpu_manager.go:414] "containerMap: removing container" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" Oct 30 19:09:36.059298 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.059281 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" Oct 30 19:09:36.059449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.059439 5959 cpu_manager.go:414] "containerMap: removing container" podUID="3d484e54-c577-47a7-a127-33548ae935bf" containerName="payload-pre-processing" Oct 30 19:09:36.059522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.059512 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d484e54-c577-47a7-a127-33548ae935bf" containerName="payload-pre-processing" Oct 30 19:09:36.059853 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.059836 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d881827-4f39-4fa5-967c-3d725e98e4f4" containerName="main" Oct 30 19:09:36.059972 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.059962 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d484e54-c577-47a7-a127-33548ae935bf" containerName="payload-pre-processing" Oct 30 19:09:36.062683 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.062661 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d484e54-c577-47a7-a127-33548ae935bf-kube-api-access-rn8bq\" (UniqueName: \"kubernetes.io/projected/3d484e54-c577-47a7-a127-33548ae935bf-kube-api-access-rn8bq\") pod \"3d484e54-c577-47a7-a127-33548ae935bf\" (UID: \"3d484e54-c577-47a7-a127-33548ae935bf\") " Oct 30 19:09:36.063488 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.063465 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d484e54-c577-47a7-a127-33548ae935bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d484e54-c577-47a7-a127-33548ae935bf-plugins-config-volume\") pod \"3d484e54-c577-47a7-a127-33548ae935bf\" (UID: \"3d484e54-c577-47a7-a127-33548ae935bf\") " Oct 30 19:09:36.075138 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.075088 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n"] Oct 30 19:09:36.075319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.069966 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.075458 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.064155 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d484e54-c577-47a7-a127-33548ae935bf-plugins-config-volume" pod "3d484e54-c577-47a7-a127-33548ae935bf" (UID: "3d484e54-c577-47a7-a127-33548ae935bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:36.075458 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.066132 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d484e54-c577-47a7-a127-33548ae935bf-kube-api-access-rn8bq" pod "3d484e54-c577-47a7-a127-33548ae935bf" (UID: "3d484e54-c577-47a7-a127-33548ae935bf"). InnerVolumeSpecName "kube-api-access-rn8bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:36.076508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.076484 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d484e54-c577-47a7-a127-33548ae935bf-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:36.076703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.076512 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8bq\" (UniqueName: \"kubernetes.io/projected/3d484e54-c577-47a7-a127-33548ae935bf-kube-api-access-rn8bq\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:36.178031 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.177981 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0621de4b-186a-4f15-8d5b-79146a55c322-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.178224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.178062 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnkw\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-kube-api-access-zvnkw\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.178224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.178203 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.178310 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.178270 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.178410 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.178388 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0621de4b-186a-4f15-8d5b-79146a55c322-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.178463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.178431 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.178550 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.178534 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.178682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.178569 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.178759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.178729 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280057 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.279962 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280057 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280014 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280306 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280090 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280306 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280218 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0621de4b-186a-4f15-8d5b-79146a55c322-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280306 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280284 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnkw\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-kube-api-access-zvnkw\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280525 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280413 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280525 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280428 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280525 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280462 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280711 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280588 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280778 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280745 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0621de4b-186a-4f15-8d5b-79146a55c322-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280842 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280799 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280901 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280861 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.280965 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.280943 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.281022 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.281001 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0621de4b-186a-4f15-8d5b-79146a55c322-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.282580 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.282549 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.283225 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.283207 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0621de4b-186a-4f15-8d5b-79146a55c322-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.290856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.290827 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnkw\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-kube-api-access-zvnkw\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.291280 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.291258 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.405656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.405593 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:36.723806 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:09:36.723764 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0621de4b_186a_4f15_8d5b_79146a55c322.slice/crio-6ac65223c81d29799ba0ec592cc3b5f31e64509f7a39ebc80c3ea34a95cd6910 WatchSource:0}: Error finding container 6ac65223c81d29799ba0ec592cc3b5f31e64509f7a39ebc80c3ea34a95cd6910: Status 404 returned error can't find the container with id 6ac65223c81d29799ba0ec592cc3b5f31e64509f7a39ebc80c3ea34a95cd6910 Oct 30 19:09:36.724160 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.723955 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n"] Oct 30 19:09:36.727064 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.727029 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:36.727147 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.727104 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:36.727147 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.727135 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:36.938205 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.938165 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" event={"ID":"0621de4b-186a-4f15-8d5b-79146a55c322","Type":"ContainerStarted","Data":"9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878"} Oct 30 19:09:36.938415 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.938239 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" event={"ID":"0621de4b-186a-4f15-8d5b-79146a55c322","Type":"ContainerStarted","Data":"6ac65223c81d29799ba0ec592cc3b5f31e64509f7a39ebc80c3ea34a95cd6910"} Oct 30 19:09:36.944385 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.944340 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" event={"ID":"3d484e54-c577-47a7-a127-33548ae935bf","Type":"ContainerDied","Data":"37dc3ed50b684270d8c966699c01e7bbaf78129c5062838cbc2dbf0268fd3112"} Oct 30 19:09:36.944583 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.944419 5959 scope.go:122] "RemoveContainer" containerID="3c2b48e7b36412a348a762dfda3762f9b918376d890b0073d89d68e7dbbc597a" Oct 30 19:09:36.944740 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.944594 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs" Oct 30 19:09:36.978678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.975561 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" podStartSLOduration=8.975542557 podStartE2EDuration="8.975542557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:36.971968436 +0000 UTC m=+1957.300379822" watchObservedRunningTime="2026-10-30 19:09:36.975542557 +0000 UTC m=+1957.303953944" Oct 30 19:09:36.996975 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:36.996908 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs"] Oct 30 19:09:37.007022 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:37.006989 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c164d6-6d4885c779-255gs"] Oct 30 19:09:37.406492 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:37.406458 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:37.408252 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:37.408219 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:09:37.408418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:37.408299 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:09:37.819685 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:37.816749 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d484e54-c577-47a7-a127-33548ae935bf" path="/var/lib/kubelet/pods/3d484e54-c577-47a7-a127-33548ae935bf/volumes" Oct 30 19:09:38.412199 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:38.412162 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:09:38.412541 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:38.412508 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:09:39.417631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:39.417131 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:09:39.417631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:39.417201 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:09:40.421224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:40.421165 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:09:40.421838 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:40.421720 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:09:41.415794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:41.415750 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:09:41.416133 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:41.416089 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:09:42.415951 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:42.415911 5959 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:09:42.417319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:42.417284 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:09:43.499897 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:43.499696 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:43.500483 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:43.500221 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:43.501673 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:43.501577 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:43.888540 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:43.888487 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf"] Oct 30 19:09:43.888823 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:43.888778 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" containerID="cri-o://5bd0787b60598f34e7ce78f75eff6e8b38c68fdcded69a1d6fbd2c33af80b2cf" gracePeriod=30 Oct 30 19:09:43.899856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:43.899796 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc"] Oct 30 19:09:43.900418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:43.900333 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" containerID="cri-o://314bb93864e8cfdf1d5dc9bd0cda27245c036cdd22b0630ac1d2d2dee49f1fd2" gracePeriod=30 Oct 30 19:09:43.907150 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:43.907111 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n"] Oct 30 19:09:44.095637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:44.094958 5959 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" secret="" err="secret \"e2e-autores-c164d6-openshift-default-dockercfg-c42sr\" not found" Oct 30 19:09:45.083539 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:45.083481 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" containerID="cri-o://9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878" gracePeriod=30 Oct 30 19:09:49.125092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.124942 5959 generic.go:356] "Generic (PLEG): container finished" podID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerID="5bd0787b60598f34e7ce78f75eff6e8b38c68fdcded69a1d6fbd2c33af80b2cf" exitCode=0 Oct 30 19:09:49.125092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.125036 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" event={"ID":"ed381eeb-43ce-4084-9ee7-1436cee51197","Type":"ContainerDied","Data":"5bd0787b60598f34e7ce78f75eff6e8b38c68fdcded69a1d6fbd2c33af80b2cf"} Oct 30 19:09:49.126899 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.126869 5959 generic.go:356] "Generic (PLEG): container finished" podID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerID="314bb93864e8cfdf1d5dc9bd0cda27245c036cdd22b0630ac1d2d2dee49f1fd2" exitCode=0 Oct 30 19:09:49.127004 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.126947 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" event={"ID":"69fce9c0-114c-4aae-8ccf-33720a9cd1e3","Type":"ContainerDied","Data":"314bb93864e8cfdf1d5dc9bd0cda27245c036cdd22b0630ac1d2d2dee49f1fd2"} Oct 30 19:09:49.227733 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.227699 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:49.258434 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.258403 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:49.268626 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.268584 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5"] Oct 30 19:09:49.269518 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.269496 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" Oct 30 19:09:49.269518 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.269519 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" Oct 30 19:09:49.269731 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.269630 5959 cpu_manager.go:414] "containerMap: removing container" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" Oct 30 19:09:49.269731 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.269639 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" Oct 30 19:09:49.269848 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.269833 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" containerName="istio-proxy" Oct 30 19:09:49.269904 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.269858 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" containerName="istio-proxy" Oct 30 19:09:49.273956 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.273931 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.294292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.294262 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-37f2b8-openshift-default-dockercfg-vdhkg\"" Oct 30 19:09:49.295123 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.295099 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5"] Oct 30 19:09:49.372177 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.372138 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-podinfo\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.372514 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.372244 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-kube-api-access-jsm5d\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-kube-api-access-jsm5d\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.372514 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.372280 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-data\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.372514 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.372390 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-certs\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.373284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373029 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-credential-socket\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.373284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373076 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istiod-ca-cert\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.373284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373103 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-envoy\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.373284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373206 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-token\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-token\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.373284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373276 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-token\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-token\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.373284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373275 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-certs" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.373721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373304 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-data" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.373721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373343 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed381eeb-43ce-4084-9ee7-1436cee51197-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed381eeb-43ce-4084-9ee7-1436cee51197-istiod-ca-cert\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.373721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373408 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-data\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.373721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373493 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-credential-socket\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.373721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373533 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-socket\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.373721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373589 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istiod-ca-cert" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:49.373721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373597 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-podinfo\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373760 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-kube-api-access-mzw75\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-kube-api-access-mzw75\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373800 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-certs\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373920 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-socket\") pod \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\" (UID: \"69fce9c0-114c-4aae-8ccf-33720a9cd1e3\") " Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.373986 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-envoy\") pod \"ed381eeb-43ce-4084-9ee7-1436cee51197\" (UID: \"ed381eeb-43ce-4084-9ee7-1436cee51197\") " Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.374469 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-socket\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.374741 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-token\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.374865 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842w5\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-kube-api-access-842w5\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.374920 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-certs\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375072 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-envoy\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375114 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-data\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375263 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-podinfo\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375396 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istiod-ca-cert\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375435 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-credential-socket\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375598 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375634 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375651 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.375895 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-credential-socket" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.376255 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-podinfo" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.377192 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-data" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.377262 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-certs" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.377741 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed381eeb-43ce-4084-9ee7-1436cee51197-istiod-ca-cert" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.377871 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-socket" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.377895 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-credential-socket" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.378656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.378325 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-socket" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.380049 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.378813 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-kube-api-access-mzw75" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "kube-api-access-mzw75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:49.380401 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.380373 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-token" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:49.380858 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.380833 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-envoy" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.381617 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.381582 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-token" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:49.382084 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.382056 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-kube-api-access-jsm5d" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "kube-api-access-jsm5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:49.382538 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.382505 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-podinfo" pod "69fce9c0-114c-4aae-8ccf-33720a9cd1e3" (UID: "69fce9c0-114c-4aae-8ccf-33720a9cd1e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:09:49.382726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.382706 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-envoy" pod "ed381eeb-43ce-4084-9ee7-1436cee51197" (UID: "ed381eeb-43ce-4084-9ee7-1436cee51197"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.490514 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-envoy\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.490580 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-data\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.490697 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-podinfo\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.490786 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istiod-ca-cert\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.490819 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-credential-socket\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.490939 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-socket\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491046 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-token\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491132 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842w5\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-kube-api-access-842w5\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491176 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-certs\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491329 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed381eeb-43ce-4084-9ee7-1436cee51197-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491347 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491362 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491376 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491391 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491405 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzw75\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-kube-api-access-mzw75\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491420 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491433 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491445 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491460 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491473 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsm5d\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-kube-api-access-jsm5d\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491487 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491499 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491513 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fce9c0-114c-4aae-8ccf-33720a9cd1e3-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491525 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed381eeb-43ce-4084-9ee7-1436cee51197-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.491980 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-certs\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.492719 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-credential-socket\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.494682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.492972 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-data\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.496941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.495163 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-socket\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.496941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.496167 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istiod-ca-cert\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.500379 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.499747 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-envoy\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.500379 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.500298 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-podinfo\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.505070 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.504991 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-token\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.505070 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.505043 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-842w5\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-kube-api-access-842w5\") pod \"e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.607224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.607184 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:49.903985 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:09:49.903957 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36f4f6a_e4fd_48e8_9e96_04d5a65fc49d.slice/crio-86b80675215b46b083aa2c1627c3e2e958157bbda6621d55afbfa4f7c0ce9693 WatchSource:0}: Error finding container 86b80675215b46b083aa2c1627c3e2e958157bbda6621d55afbfa4f7c0ce9693: Status 404 returned error can't find the container with id 86b80675215b46b083aa2c1627c3e2e958157bbda6621d55afbfa4f7c0ce9693 Oct 30 19:09:49.904522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.904497 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5"] Oct 30 19:09:49.907163 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.907113 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:49.907293 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.907206 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:49.907293 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:49.907259 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:09:50.175499 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.174758 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" event={"ID":"ed381eeb-43ce-4084-9ee7-1436cee51197","Type":"ContainerDied","Data":"d9709787200fa581caeea4ef32aa878a2d319adbf18e1f3c97d13bafac901dd6"} Oct 30 19:09:50.175499 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.174847 5959 scope.go:122] "RemoveContainer" containerID="5bd0787b60598f34e7ce78f75eff6e8b38c68fdcded69a1d6fbd2c33af80b2cf" Oct 30 19:09:50.175499 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.175032 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf" Oct 30 19:09:50.177614 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.177582 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" Oct 30 19:09:50.178250 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.177687 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc" event={"ID":"69fce9c0-114c-4aae-8ccf-33720a9cd1e3","Type":"ContainerDied","Data":"af2f24d1f64c2e1ba50a46d221223cd127f03e756c31dcb60f619f0cf3075468"} Oct 30 19:09:50.181890 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.181859 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" event={"ID":"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d","Type":"ContainerStarted","Data":"d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050"} Oct 30 19:09:50.182021 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.181916 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" event={"ID":"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d","Type":"ContainerStarted","Data":"86b80675215b46b083aa2c1627c3e2e958157bbda6621d55afbfa4f7c0ce9693"} Oct 30 19:09:50.208125 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.208099 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf"] Oct 30 19:09:50.216281 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.216256 5959 scope.go:122] "RemoveContainer" containerID="314bb93864e8cfdf1d5dc9bd0cda27245c036cdd22b0630ac1d2d2dee49f1fd2" Oct 30 19:09:50.218674 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.218458 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-jh2jf"] Oct 30 19:09:50.227759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.227732 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc"] Oct 30 19:09:50.231912 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.231885 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-27pwc"] Oct 30 19:09:50.247800 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.247745 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" podStartSLOduration=4.247728732 podStartE2EDuration="4.247728732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:50.244789533 +0000 UTC m=+1970.573200917" watchObservedRunningTime="2026-10-30 19:09:50.247728732 +0000 UTC m=+1970.576140126" Oct 30 19:09:50.489686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.488365 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:50.608261 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.608204 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:50.618118 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618083 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-kube-api-access-zvnkw\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-kube-api-access-zvnkw\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618336 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618130 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-socket\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618336 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618245 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-credential-socket\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618336 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618305 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-data\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618512 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618336 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-istio-token\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-istio-token\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618512 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618426 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-envoy\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618512 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618446 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-socket" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.618512 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618461 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0621de4b-186a-4f15-8d5b-79146a55c322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0621de4b-186a-4f15-8d5b-79146a55c322-istiod-ca-cert\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618556 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-credential-socket" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.618761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618690 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-certs\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618932 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618819 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0621de4b-186a-4f15-8d5b-79146a55c322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0621de4b-186a-4f15-8d5b-79146a55c322-istio-podinfo\") pod \"0621de4b-186a-4f15-8d5b-79146a55c322\" (UID: \"0621de4b-186a-4f15-8d5b-79146a55c322\") " Oct 30 19:09:50.618932 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.618827 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0621de4b-186a-4f15-8d5b-79146a55c322-istiod-ca-cert" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:50.619127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.619109 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-data" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.619301 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.619284 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-certs" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.619920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.619826 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:50.620032 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.620003 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:50.620032 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.620026 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:50.620162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.620041 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0621de4b-186a-4f15-8d5b-79146a55c322-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:50.620162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.620055 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:50.620803 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.620774 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-kube-api-access-zvnkw" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "kube-api-access-zvnkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:50.621435 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.621417 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-envoy" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.621563 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.621427 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-istio-token" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:50.621563 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.621555 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0621de4b-186a-4f15-8d5b-79146a55c322-istio-podinfo" pod "0621de4b-186a-4f15-8d5b-79146a55c322" (UID: "0621de4b-186a-4f15-8d5b-79146a55c322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:09:50.721378 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.721288 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:50.721378 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.721326 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0621de4b-186a-4f15-8d5b-79146a55c322-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:50.721378 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.721336 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0621de4b-186a-4f15-8d5b-79146a55c322-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:50.721378 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:50.721346 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvnkw\" (UniqueName: \"kubernetes.io/projected/0621de4b-186a-4f15-8d5b-79146a55c322-kube-api-access-zvnkw\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:51.191941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.191907 5959 generic.go:356] "Generic (PLEG): container finished" podID="0621de4b-186a-4f15-8d5b-79146a55c322" containerID="9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878" exitCode=0 Oct 30 19:09:51.192454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.191964 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" Oct 30 19:09:51.192454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.191986 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" event={"ID":"0621de4b-186a-4f15-8d5b-79146a55c322","Type":"ContainerDied","Data":"9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878"} Oct 30 19:09:51.192454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.192055 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n" event={"ID":"0621de4b-186a-4f15-8d5b-79146a55c322","Type":"ContainerDied","Data":"6ac65223c81d29799ba0ec592cc3b5f31e64509f7a39ebc80c3ea34a95cd6910"} Oct 30 19:09:51.192454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.192087 5959 scope.go:122] "RemoveContainer" containerID="9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878" Oct 30 19:09:51.206340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.206321 5959 scope.go:122] "RemoveContainer" containerID="9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878" Oct 30 19:09:51.206725 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:09:51.206703 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878\": container with ID starting with 9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878 not found: ID does not exist" containerID="9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878" Oct 30 19:09:51.206784 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.206734 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878"} err="failed to get container status \"9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878\": rpc error: code = NotFound desc = could not find container \"9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878\": container with ID starting with 9ac50bf684899a5fc81977f834e9c94c187df8d3bba03a036ede89e28f356878 not found: ID does not exist" Oct 30 19:09:51.219736 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.219699 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n"] Oct 30 19:09:51.224013 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.223985 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4z45n"] Oct 30 19:09:51.614333 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.614241 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:51.827075 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.827021 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" path="/var/lib/kubelet/pods/0621de4b-186a-4f15-8d5b-79146a55c322/volumes" Oct 30 19:09:51.830127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.827797 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fce9c0-114c-4aae-8ccf-33720a9cd1e3" path="/var/lib/kubelet/pods/69fce9c0-114c-4aae-8ccf-33720a9cd1e3/volumes" Oct 30 19:09:51.830127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:51.828433 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed381eeb-43ce-4084-9ee7-1436cee51197" path="/var/lib/kubelet/pods/ed381eeb-43ce-4084-9ee7-1436cee51197/volumes" Oct 30 19:09:52.205779 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:52.205747 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:52.207770 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:09:52.207747 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:09:57.781984 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:09:57.781940 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:10:00.739413 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.739376 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94"] Oct 30 19:10:00.740548 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.740278 5959 cpu_manager.go:414] "containerMap: removing container" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" Oct 30 19:10:00.740548 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.740292 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" Oct 30 19:10:00.740548 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.740484 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0621de4b-186a-4f15-8d5b-79146a55c322" containerName="istio-proxy" Oct 30 19:10:00.745971 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.745941 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.762171 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.762140 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94"] Oct 30 19:10:00.860883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.860845 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.861258 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.861204 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.861342 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.861276 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.861421 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.861391 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bdl\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-kube-api-access-p6bdl\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.861527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.861507 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.861677 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.861595 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.861758 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.861738 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.861860 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.861847 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.861942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.861925 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.963466 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.963424 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.963827 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.963800 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.963945 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.963866 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.963945 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.963912 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bdl\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-kube-api-access-p6bdl\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.963945 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.963924 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.964130 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964002 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.964130 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964083 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.964250 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964180 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.964334 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964313 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.964486 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964412 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.964624 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964586 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.964969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964938 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.964969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964965 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.965119 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.964944 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.966895 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.966873 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.966976 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.966947 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.972412 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.972383 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:00.972540 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:00.972520 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bdl\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-kube-api-access-p6bdl\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:01.080383 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:01.080290 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:01.413776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:01.413737 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94"] Oct 30 19:10:01.518809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:01.517582 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:10:01.518809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:01.517712 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:10:01.518809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:01.517757 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:10:02.529317 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:02.529281 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" event={"ID":"78abee8b-e5f9-4ff4-be79-ca766fbb26c0","Type":"ContainerStarted","Data":"b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661"} Oct 30 19:10:02.529838 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:02.529341 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" event={"ID":"78abee8b-e5f9-4ff4-be79-ca766fbb26c0","Type":"ContainerStarted","Data":"86eb5b5707f14f8cc445a519627d9fcae7019c7170216afae7b1a639d7a1886b"} Oct 30 19:10:02.591969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:02.591910 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" podStartSLOduration=2.591890268 podStartE2EDuration="2.591890268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:02.590457723 +0000 UTC m=+1982.918869108" watchObservedRunningTime="2026-10-30 19:10:02.591890268 +0000 UTC m=+1982.920301714" Oct 30 19:10:03.081385 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:03.081344 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:03.083038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:03.083003 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:10:03.083219 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:03.083071 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 19:10:04.082130 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:04.082089 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:10:04.082791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:04.082757 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 19:10:05.127632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:05.126847 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:10:05.127632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:05.126988 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 19:10:06.034578 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.034539 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658"] Oct 30 19:10:06.040013 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.039949 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.046783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.046754 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"agc-cred-b58707117777\"" Oct 30 19:10:06.047258 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.046803 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f3d6a5\"" Oct 30 19:10:06.047505 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.046821 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"routing-overlay\"" Oct 30 19:10:06.047626 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.046903 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"openshift-service-ca.crt\"" Oct 30 19:10:06.047737 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.046967 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"kube-root-ca.crt\"" Oct 30 19:10:06.047824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.047021 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"agc-cred-dc5412cc868a\"" Oct 30 19:10:06.047907 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.047088 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"payload-processing-external-model-e2e--eb883441-dockercfg-2ql8r\"" Oct 30 19:10:06.055109 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.054258 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658"] Oct 30 19:10:06.080768 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.080729 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:10:06.081081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.081053 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 19:10:06.151640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.143907 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cs4g\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-kube-api-access-6cs4g\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.152279 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.152249 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.170990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.152863 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.171332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.171298 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.302687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.283776 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.385789 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.385733 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.413905 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.413865 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cs4g\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-kube-api-access-6cs4g\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.414187 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.414165 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.415639 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.415546 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.452247 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.304735 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.454011 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.453979 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.479463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.479370 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cs4g\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-kube-api-access-6cs4g\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.665786 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.665751 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:06.875701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.872644 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx"] Oct 30 19:10:06.880920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.880893 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:06.882127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.882104 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx"] Oct 30 19:10:06.888914 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.888361 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a477d\"/\"agc-cred-94828ab1ab88\"" Oct 30 19:10:06.888914 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.888487 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a477d\"/\"payload-processing-external-model-e2e--27d7d667-dockercfg-ffnd8\"" Oct 30 19:10:06.888914 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.888552 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a477d\"/\"agc-cred-bd948f818f73\"" Oct 30 19:10:06.889133 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.888929 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a477d\"/\"routing-overlay\"" Oct 30 19:10:06.889133 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.889040 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a477d\"/\"openshift-service-ca.crt\"" Oct 30 19:10:06.889874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.889532 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a477d\"/\"kube-root-ca.crt\"" Oct 30 19:10:06.889874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.889658 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a477d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4a477d\"" Oct 30 19:10:06.994415 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.994372 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:06.995189 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.995150 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:06.995335 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.995277 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:06.995513 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:06.995487 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgq29\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-kube-api-access-dgq29\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.089775 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.089734 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:10:07.100794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.100747 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 19:10:07.144558 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.144505 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq29\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-kube-api-access-dgq29\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.144795 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.144709 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.145038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.145006 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.145113 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.145086 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.145785 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.145757 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.146284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.146005 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.149305 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.149266 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658"] Oct 30 19:10:07.149781 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.149747 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.150304 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:10:07.150270 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e0e9ca_04c2_461d_9749_7a31da9847cb.slice/crio-2c2f421de3f76a9fa7e7fd0bfcdeaa219e78a682f56b49ce50b96e467a1c2050 WatchSource:0}: Error finding container 2c2f421de3f76a9fa7e7fd0bfcdeaa219e78a682f56b49ce50b96e467a1c2050: Status 404 returned error can't find the container with id 2c2f421de3f76a9fa7e7fd0bfcdeaa219e78a682f56b49ce50b96e467a1c2050 Oct 30 19:10:07.154994 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.154967 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq29\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-kube-api-access-dgq29\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.261476 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.261438 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:07.579725 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.579628 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" event={"ID":"87e0e9ca-04c2-461d-9749-7a31da9847cb","Type":"ContainerStarted","Data":"89e31ab7c0569ac81517e4be4100d9139f640cafd58ac84cecbc5fcdd81e52f7"} Oct 30 19:10:07.579725 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.579691 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:07.579725 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.579706 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" event={"ID":"87e0e9ca-04c2-461d-9749-7a31da9847cb","Type":"ContainerStarted","Data":"2c2f421de3f76a9fa7e7fd0bfcdeaa219e78a682f56b49ce50b96e467a1c2050"} Oct 30 19:10:07.588682 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:10:07.588651 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2eeb47_ef9d_4ab2_9d92_71d5045e7428.slice/crio-20f31fcef6d95b5688076ce2218bcdbc918a97f0e9e27161933752be57a051d6 WatchSource:0}: Error finding container 20f31fcef6d95b5688076ce2218bcdbc918a97f0e9e27161933752be57a051d6: Status 404 returned error can't find the container with id 20f31fcef6d95b5688076ce2218bcdbc918a97f0e9e27161933752be57a051d6 Oct 30 19:10:07.588922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:07.588898 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx"] Oct 30 19:10:08.081335 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:08.081297 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:10:08.081553 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:08.081351 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 19:10:08.592488 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:08.592449 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" event={"ID":"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428","Type":"ContainerStarted","Data":"ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252"} Oct 30 19:10:08.593058 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:08.592512 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" event={"ID":"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428","Type":"ContainerStarted","Data":"20f31fcef6d95b5688076ce2218bcdbc918a97f0e9e27161933752be57a051d6"} Oct 30 19:10:08.593058 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:08.592778 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:08.615452 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:08.615385 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" podStartSLOduration=2.615370327 podStartE2EDuration="2.615370327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:08.612840685 +0000 UTC m=+1988.941252070" watchObservedRunningTime="2026-10-30 19:10:08.615370327 +0000 UTC m=+1988.943781715" Oct 30 19:10:08.617871 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:08.617825 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" podStartSLOduration=2.617808436 podStartE2EDuration="2.617808436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:07.623009949 +0000 UTC m=+1987.951421336" watchObservedRunningTime="2026-10-30 19:10:08.617808436 +0000 UTC m=+1988.946219815" Oct 30 19:10:09.086869 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:09.086836 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:09.087378 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:09.087359 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:09.088210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:09.088190 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:10:18.596447 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:18.596368 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:10:19.604358 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:19.604331 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:10:27.783973 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:10:27.783922 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:10:33.814159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:33.814119 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:10:47.126726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:47.115274 5959 generic.go:356] "Generic (PLEG): container finished" podID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerID="d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050" exitCode=0 Oct 30 19:10:47.126726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:47.115498 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" event={"ID":"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d","Type":"ContainerDied","Data":"d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050"} Oct 30 19:10:47.126726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:47.116120 5959 scope.go:122] "RemoveContainer" containerID="d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050" Oct 30 19:10:47.131153 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:47.131105 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:10:47.131324 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:47.131211 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:10:47.131324 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:47.131270 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:10:48.139449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:48.139404 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" event={"ID":"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d","Type":"ContainerStarted","Data":"6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f"} Oct 30 19:10:48.612832 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:48.612739 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:10:48.613232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:48.613209 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 19:10:48.613525 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:48.613497 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 19:10:49.620365 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:49.620311 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 19:10:49.621107 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:49.621072 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 19:10:50.613063 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:50.613025 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 19:10:50.613368 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:50.613338 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 19:10:51.612373 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:51.611744 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 19:10:51.612373 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:51.611824 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 19:10:52.223016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:52.221757 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:10:52.616693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:52.616369 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 19:10:52.616693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:52.616451 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 19:10:53.607732 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:53.607691 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 19:10:53.607974 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:53.607760 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 19:10:54.613871 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:54.613838 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:10:54.615055 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:10:54.615030 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:10:57.811440 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:10:57.811392 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:11:27.782587 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:27.782547 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:11:31.869174 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:31.869143 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a477d/agc-cred-94828ab1ab88: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:31.869174 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:31.869175 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:31.869804 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:31.869260 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials podName:2a2eeb47-ef9d-4ab2-9d92-71d5045e7428 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:32.369242598 +0000 UTC m=+2072.697653987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428") : secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:32.379377 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:32.379341 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a477d/agc-cred-94828ab1ab88: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:32.379377 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:32.379385 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:32.379666 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:32.379459 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials podName:2a2eeb47-ef9d-4ab2-9d92-71d5045e7428 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:33.379439297 +0000 UTC m=+2073.707850665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428") : secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:33.139385 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:11:33.139350 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:11:33.403672 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:33.403558 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a477d/agc-cred-94828ab1ab88: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:33.404024 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:33.404006 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:33.404208 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:33.404187 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials podName:2a2eeb47-ef9d-4ab2-9d92-71d5045e7428 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:35.404162643 +0000 UTC m=+2075.732574010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428") : secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:35.425165 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:35.425082 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a477d/agc-cred-94828ab1ab88: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:35.425787 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:35.425768 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:35.426034 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:35.426008 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials podName:2a2eeb47-ef9d-4ab2-9d92-71d5045e7428 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:39.425982507 +0000 UTC m=+2079.754393891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428") : secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:39.530228 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:39.530178 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a477d/agc-cred-94828ab1ab88: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:39.530228 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:39.530222 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:39.530888 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:39.530291 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials podName:2a2eeb47-ef9d-4ab2-9d92-71d5045e7428 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:47.530272922 +0000 UTC m=+2087.858684285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428") : secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:47.628657 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:47.628555 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a477d/agc-cred-94828ab1ab88: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:47.628657 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:47.628592 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx: secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:47.629142 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:47.628666 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials podName:2a2eeb47-ef9d-4ab2-9d92-71d5045e7428 nodeName:}" failed. No retries permitted until 2026-10-30 19:12:03.628648972 +0000 UTC m=+2103.957060336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428") : secret "agc-cred-94828ab1ab88" not found Oct 30 19:11:57.784184 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:11:57.784140 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:12:03.644162 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:03.644124 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a477d/agc-cred-94828ab1ab88: secret "agc-cred-94828ab1ab88" not found Oct 30 19:12:03.644873 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:03.644850 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx: secret "agc-cred-94828ab1ab88" not found Oct 30 19:12:03.645022 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:03.644944 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials podName:2a2eeb47-ef9d-4ab2-9d92-71d5045e7428 nodeName:}" failed. No retries permitted until 2026-10-30 19:12:35.644913802 +0000 UTC m=+2135.973325194 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428") : secret "agc-cred-94828ab1ab88" not found Oct 30 19:12:18.599151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.599115 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr"] Oct 30 19:12:18.604902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.604878 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.610103 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.610076 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:18.610340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.610222 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\"" Oct 30 19:12:18.614210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.614183 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr"] Oct 30 19:12:18.701811 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.701776 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88531eb4-beff-48eb-b123-8da03308abca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.702143 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.702113 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-464qw\" (UniqueName: \"kubernetes.io/projected/88531eb4-beff-48eb-b123-8da03308abca-kube-api-access-464qw\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.702289 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.702166 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88531eb4-beff-48eb-b123-8da03308abca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.804357 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.804320 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88531eb4-beff-48eb-b123-8da03308abca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.804930 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.804894 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88531eb4-beff-48eb-b123-8da03308abca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.805087 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.804953 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464qw\" (UniqueName: \"kubernetes.io/projected/88531eb4-beff-48eb-b123-8da03308abca-kube-api-access-464qw\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.805490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.805465 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88531eb4-beff-48eb-b123-8da03308abca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.807106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.807083 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88531eb4-beff-48eb-b123-8da03308abca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.812997 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.812967 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-464qw\" (UniqueName: \"kubernetes.io/projected/88531eb4-beff-48eb-b123-8da03308abca-kube-api-access-464qw\") pod \"maas-api-revoke-sub-9593db2b-wz4dr\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:18.923197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:18.923154 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:19.217960 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:12:19.217922 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88531eb4_beff_48eb_b123_8da03308abca.slice/crio-245937ee2a2fa252035e7fe35506dc2d5913cad0f8534cfba163aebe82bd16ad WatchSource:0}: Error finding container 245937ee2a2fa252035e7fe35506dc2d5913cad0f8534cfba163aebe82bd16ad: Status 404 returned error can't find the container with id 245937ee2a2fa252035e7fe35506dc2d5913cad0f8534cfba163aebe82bd16ad Oct 30 19:12:19.218115 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:19.218071 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr"] Oct 30 19:12:20.231279 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:20.231241 5959 generic.go:356] "Generic (PLEG): container finished" podID="88531eb4-beff-48eb-b123-8da03308abca" containerID="a5e1f7e45471b8664c5da68a58c5840cceba8d5131941dfb447f0641069841f1" exitCode=0 Oct 30 19:12:20.231802 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:20.231292 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" event={"ID":"88531eb4-beff-48eb-b123-8da03308abca","Type":"ContainerDied","Data":"a5e1f7e45471b8664c5da68a58c5840cceba8d5131941dfb447f0641069841f1"} Oct 30 19:12:20.231802 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:20.231364 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" event={"ID":"88531eb4-beff-48eb-b123-8da03308abca","Type":"ContainerStarted","Data":"245937ee2a2fa252035e7fe35506dc2d5913cad0f8534cfba163aebe82bd16ad"} Oct 30 19:12:21.456340 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.456310 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:21.571722 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.571621 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88531eb4-beff-48eb-b123-8da03308abca-kube-api-access-464qw\" (UniqueName: \"kubernetes.io/projected/88531eb4-beff-48eb-b123-8da03308abca-kube-api-access-464qw\") pod \"88531eb4-beff-48eb-b123-8da03308abca\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " Oct 30 19:12:21.572030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.572005 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88531eb4-beff-48eb-b123-8da03308abca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88531eb4-beff-48eb-b123-8da03308abca-maas-api-openshift-ca\") pod \"88531eb4-beff-48eb-b123-8da03308abca\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " Oct 30 19:12:21.572190 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.572171 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88531eb4-beff-48eb-b123-8da03308abca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88531eb4-beff-48eb-b123-8da03308abca-maas-api-xks-ca\") pod \"88531eb4-beff-48eb-b123-8da03308abca\" (UID: \"88531eb4-beff-48eb-b123-8da03308abca\") " Oct 30 19:12:21.576303 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.576265 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88531eb4-beff-48eb-b123-8da03308abca-maas-api-xks-ca" pod "88531eb4-beff-48eb-b123-8da03308abca" (UID: "88531eb4-beff-48eb-b123-8da03308abca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:21.576794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.576770 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88531eb4-beff-48eb-b123-8da03308abca-maas-api-openshift-ca" pod "88531eb4-beff-48eb-b123-8da03308abca" (UID: "88531eb4-beff-48eb-b123-8da03308abca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:21.577393 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.577359 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88531eb4-beff-48eb-b123-8da03308abca-kube-api-access-464qw" pod "88531eb4-beff-48eb-b123-8da03308abca" (UID: "88531eb4-beff-48eb-b123-8da03308abca"). InnerVolumeSpecName "kube-api-access-464qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:21.675331 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.675293 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-464qw\" (UniqueName: \"kubernetes.io/projected/88531eb4-beff-48eb-b123-8da03308abca-kube-api-access-464qw\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:21.675331 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.675324 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88531eb4-beff-48eb-b123-8da03308abca-maas-api-openshift-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:21.675331 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:21.675336 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88531eb4-beff-48eb-b123-8da03308abca-maas-api-xks-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:22.269017 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:22.268988 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" Oct 30 19:12:22.269192 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:22.269016 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr" event={"ID":"88531eb4-beff-48eb-b123-8da03308abca","Type":"ContainerDied","Data":"245937ee2a2fa252035e7fe35506dc2d5913cad0f8534cfba163aebe82bd16ad"} Oct 30 19:12:22.269192 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:22.269067 5959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245937ee2a2fa252035e7fe35506dc2d5913cad0f8534cfba163aebe82bd16ad" Oct 30 19:12:26.295920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:26.295886 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:12:27.782654 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:27.782592 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:12:29.213665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:29.213628 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s"] Oct 30 19:12:30.102058 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.102026 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89"] Oct 30 19:12:30.102943 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.102927 5959 cpu_manager.go:414] "containerMap: removing container" podUID="88531eb4-beff-48eb-b123-8da03308abca" containerName="revoke-keys" Oct 30 19:12:30.102993 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.102945 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88531eb4-beff-48eb-b123-8da03308abca" containerName="revoke-keys" Oct 30 19:12:30.103122 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.103113 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="88531eb4-beff-48eb-b123-8da03308abca" containerName="revoke-keys" Oct 30 19:12:30.107122 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.107107 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.111244 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.111222 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:30.111439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.111273 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\"" Oct 30 19:12:30.113740 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.113720 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89"] Oct 30 19:12:30.224808 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.224778 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.225319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.224949 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.225319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.224976 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zqb\" (UniqueName: \"kubernetes.io/projected/c0dad2ee-3398-4525-b9e4-fca02023f015-kube-api-access-67zqb\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.328776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.328738 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.329002 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.328985 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.329078 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.329027 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zqb\" (UniqueName: \"kubernetes.io/projected/c0dad2ee-3398-4525-b9e4-fca02023f015-kube-api-access-67zqb\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.329533 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.329504 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.331912 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.331889 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.338989 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.338957 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zqb\" (UniqueName: \"kubernetes.io/projected/c0dad2ee-3398-4525-b9e4-fca02023f015-kube-api-access-67zqb\") pod \"maas-api-revoke-sub-2ea8e386-zzh89\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.427324 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.427294 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:30.713353 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:12:30.713309 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0dad2ee_3398_4525_b9e4_fca02023f015.slice/crio-c48593085165b11992d9bc78bbcc5ccef71a3ed177d41cb650288b6e285b567b WatchSource:0}: Error finding container c48593085165b11992d9bc78bbcc5ccef71a3ed177d41cb650288b6e285b567b: Status 404 returned error can't find the container with id c48593085165b11992d9bc78bbcc5ccef71a3ed177d41cb650288b6e285b567b Oct 30 19:12:30.713527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:30.713502 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89"] Oct 30 19:12:31.354182 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:31.354095 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" event={"ID":"c0dad2ee-3398-4525-b9e4-fca02023f015","Type":"ContainerStarted","Data":"65454e5369e4ac7b39703a1bdc8be3c9e1b3cd4f224b0217dd774a6175bd86f9"} Oct 30 19:12:31.354182 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:31.354149 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" event={"ID":"c0dad2ee-3398-4525-b9e4-fca02023f015","Type":"ContainerStarted","Data":"c48593085165b11992d9bc78bbcc5ccef71a3ed177d41cb650288b6e285b567b"} Oct 30 19:12:31.374669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:31.374600 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" podStartSLOduration=1.374584368 podStartE2EDuration="1.374584368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:31.371976517 +0000 UTC m=+2131.700387903" watchObservedRunningTime="2026-10-30 19:12:31.374584368 +0000 UTC m=+2131.702995755" Oct 30 19:12:32.363356 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:32.363321 5959 generic.go:356] "Generic (PLEG): container finished" podID="c0dad2ee-3398-4525-b9e4-fca02023f015" containerID="65454e5369e4ac7b39703a1bdc8be3c9e1b3cd4f224b0217dd774a6175bd86f9" exitCode=0 Oct 30 19:12:32.363969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:32.363417 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" event={"ID":"c0dad2ee-3398-4525-b9e4-fca02023f015","Type":"ContainerDied","Data":"65454e5369e4ac7b39703a1bdc8be3c9e1b3cd4f224b0217dd774a6175bd86f9"} Oct 30 19:12:33.564624 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.564587 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:33.688739 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.688708 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-xks-ca\") pod \"c0dad2ee-3398-4525-b9e4-fca02023f015\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " Oct 30 19:12:33.688916 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.688846 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-openshift-ca\") pod \"c0dad2ee-3398-4525-b9e4-fca02023f015\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " Oct 30 19:12:33.689174 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.689150 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0dad2ee-3398-4525-b9e4-fca02023f015-kube-api-access-67zqb\" (UniqueName: \"kubernetes.io/projected/c0dad2ee-3398-4525-b9e4-fca02023f015-kube-api-access-67zqb\") pod \"c0dad2ee-3398-4525-b9e4-fca02023f015\" (UID: \"c0dad2ee-3398-4525-b9e4-fca02023f015\") " Oct 30 19:12:33.689302 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.689189 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-openshift-ca" pod "c0dad2ee-3398-4525-b9e4-fca02023f015" (UID: "c0dad2ee-3398-4525-b9e4-fca02023f015"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:33.689886 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.689868 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-openshift-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:33.691119 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.691096 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-xks-ca" pod "c0dad2ee-3398-4525-b9e4-fca02023f015" (UID: "c0dad2ee-3398-4525-b9e4-fca02023f015"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:33.691347 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.691329 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0dad2ee-3398-4525-b9e4-fca02023f015-kube-api-access-67zqb" pod "c0dad2ee-3398-4525-b9e4-fca02023f015" (UID: "c0dad2ee-3398-4525-b9e4-fca02023f015"). InnerVolumeSpecName "kube-api-access-67zqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:33.791075 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.791042 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dad2ee-3398-4525-b9e4-fca02023f015-maas-api-xks-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:33.791075 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:33.791070 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zqb\" (UniqueName: \"kubernetes.io/projected/c0dad2ee-3398-4525-b9e4-fca02023f015-kube-api-access-67zqb\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:34.214256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.214220 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" podUID="50e94bd6-1134-498a-b3d9-fab0acb86329" containerName="payload-processing" containerID="cri-o://5ff6f60a21e8056bde92a6f144500b2be5153061da4b55293b8c78c75a3950c2" gracePeriod=25 Oct 30 19:12:34.380537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.380505 5959 generic.go:356] "Generic (PLEG): container finished" podID="50e94bd6-1134-498a-b3d9-fab0acb86329" containerID="5ff6f60a21e8056bde92a6f144500b2be5153061da4b55293b8c78c75a3950c2" exitCode=0 Oct 30 19:12:34.380744 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.380575 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" event={"ID":"50e94bd6-1134-498a-b3d9-fab0acb86329","Type":"ContainerDied","Data":"5ff6f60a21e8056bde92a6f144500b2be5153061da4b55293b8c78c75a3950c2"} Oct 30 19:12:34.382141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.382119 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" Oct 30 19:12:34.382259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.382141 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89" event={"ID":"c0dad2ee-3398-4525-b9e4-fca02023f015","Type":"ContainerDied","Data":"c48593085165b11992d9bc78bbcc5ccef71a3ed177d41cb650288b6e285b567b"} Oct 30 19:12:34.382259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.382182 5959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48593085165b11992d9bc78bbcc5ccef71a3ed177d41cb650288b6e285b567b" Oct 30 19:12:34.492404 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.492381 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:12:34.525463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.525436 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r"] Oct 30 19:12:34.526750 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.526725 5959 cpu_manager.go:414] "containerMap: removing container" podUID="c0dad2ee-3398-4525-b9e4-fca02023f015" containerName="revoke-keys" Oct 30 19:12:34.526870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.526751 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0dad2ee-3398-4525-b9e4-fca02023f015" containerName="revoke-keys" Oct 30 19:12:34.526870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.526798 5959 cpu_manager.go:414] "containerMap: removing container" podUID="50e94bd6-1134-498a-b3d9-fab0acb86329" containerName="payload-processing" Oct 30 19:12:34.526870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.526807 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e94bd6-1134-498a-b3d9-fab0acb86329" containerName="payload-processing" Oct 30 19:12:34.527024 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.527012 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0dad2ee-3398-4525-b9e4-fca02023f015" containerName="revoke-keys" Oct 30 19:12:34.527063 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.527034 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e94bd6-1134-498a-b3d9-fab0acb86329" containerName="payload-processing" Oct 30 19:12:34.531953 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.531930 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.535677 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.535596 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"agc-cred-7b1a12f62240\"" Oct 30 19:12:34.539645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.539620 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r"] Oct 30 19:12:34.604718 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.604691 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-provider-credentials\") pod \"50e94bd6-1134-498a-b3d9-fab0acb86329\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " Oct 30 19:12:34.605042 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.604843 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-routing-overlay-volume\") pod \"50e94bd6-1134-498a-b3d9-fab0acb86329\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " Oct 30 19:12:34.605042 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.605007 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-plugins-config-volume\") pod \"50e94bd6-1134-498a-b3d9-fab0acb86329\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " Oct 30 19:12:34.605154 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.605052 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-kube-api-access-9njd5\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-kube-api-access-9njd5\") pod \"50e94bd6-1134-498a-b3d9-fab0acb86329\" (UID: \"50e94bd6-1134-498a-b3d9-fab0acb86329\") " Oct 30 19:12:34.605207 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.605171 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-routing-overlay-volume" pod "50e94bd6-1134-498a-b3d9-fab0acb86329" (UID: "50e94bd6-1134-498a-b3d9-fab0acb86329"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:34.605426 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.605402 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-plugins-config-volume" pod "50e94bd6-1134-498a-b3d9-fab0acb86329" (UID: "50e94bd6-1134-498a-b3d9-fab0acb86329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:34.605546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.605418 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.605659 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.605594 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.605724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.605677 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4knn\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-kube-api-access-k4knn\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.605882 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.605862 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.606170 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.606152 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:34.606233 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.606178 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e94bd6-1134-498a-b3d9-fab0acb86329-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:34.607298 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.607276 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-provider-credentials" pod "50e94bd6-1134-498a-b3d9-fab0acb86329" (UID: "50e94bd6-1134-498a-b3d9-fab0acb86329"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:34.607376 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.607346 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-kube-api-access-9njd5" pod "50e94bd6-1134-498a-b3d9-fab0acb86329" (UID: "50e94bd6-1134-498a-b3d9-fab0acb86329"). InnerVolumeSpecName "kube-api-access-9njd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:34.708155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.708126 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.708303 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.708212 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.708303 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.708237 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4knn\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-kube-api-access-k4knn\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.708415 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.708362 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.708491 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.708478 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:34.708543 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.708495 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njd5\" (UniqueName: \"kubernetes.io/projected/50e94bd6-1134-498a-b3d9-fab0acb86329-kube-api-access-9njd5\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:34.709005 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.708979 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.709085 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.709040 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.710968 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.710950 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.718888 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.718861 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4knn\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-kube-api-access-k4knn\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:34.869436 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:34.869348 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:35.168015 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.167988 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r"] Oct 30 19:12:35.169482 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:12:35.169455 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fab75b5_9db9_415e_86be_7ea8f7d5e205.slice/crio-4270abfd6534dd4120f4e794f3abe9ffd9c915532c6403d25c722f130a1b4157 WatchSource:0}: Error finding container 4270abfd6534dd4120f4e794f3abe9ffd9c915532c6403d25c722f130a1b4157: Status 404 returned error can't find the container with id 4270abfd6534dd4120f4e794f3abe9ffd9c915532c6403d25c722f130a1b4157 Oct 30 19:12:35.394353 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.393651 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" event={"ID":"5fab75b5-9db9-415e-86be-7ea8f7d5e205","Type":"ContainerStarted","Data":"6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd"} Oct 30 19:12:35.394353 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.393717 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" event={"ID":"5fab75b5-9db9-415e-86be-7ea8f7d5e205","Type":"ContainerStarted","Data":"4270abfd6534dd4120f4e794f3abe9ffd9c915532c6403d25c722f130a1b4157"} Oct 30 19:12:35.394353 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.393787 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:35.396232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.395764 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" event={"ID":"50e94bd6-1134-498a-b3d9-fab0acb86329","Type":"ContainerDied","Data":"2c5583f0e7c6ddcf1f9dfa057b266d073950a199092862cb2658d4b2d0760a04"} Oct 30 19:12:35.396232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.395824 5959 scope.go:122] "RemoveContainer" containerID="5ff6f60a21e8056bde92a6f144500b2be5153061da4b55293b8c78c75a3950c2" Oct 30 19:12:35.396232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.395995 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s" Oct 30 19:12:35.433592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.433517 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" podStartSLOduration=117.433494462 podStartE2EDuration="1m57.433494462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:35.424152274 +0000 UTC m=+2135.752563660" watchObservedRunningTime="2026-10-30 19:12:35.433494462 +0000 UTC m=+2135.761905848" Oct 30 19:12:35.458965 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.458935 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s"] Oct 30 19:12:35.464292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.464263 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28c1e9/payload-processing-external-model-e2e-extmod-b-28c1e9-6fff4ch8s"] Oct 30 19:12:35.730552 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:35.730456 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a477d/agc-cred-94828ab1ab88: secret "agc-cred-94828ab1ab88" not found Oct 30 19:12:35.730552 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:35.730486 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx: secret "agc-cred-94828ab1ab88" not found Oct 30 19:12:35.730552 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:35.730548 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials podName:2a2eeb47-ef9d-4ab2-9d92-71d5045e7428 nodeName:}" failed. No retries permitted until 2026-10-30 19:13:39.730532544 +0000 UTC m=+2200.058943906 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428") : secret "agc-cred-94828ab1ab88" not found Oct 30 19:12:35.800818 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:35.800787 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e94bd6-1134-498a-b3d9-fab0acb86329" path="/var/lib/kubelet/pods/50e94bd6-1134-498a-b3d9-fab0acb86329/volumes" Oct 30 19:12:36.929674 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:36.928816 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz"] Oct 30 19:12:39.429026 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:39.428832 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r"] Oct 30 19:12:41.930135 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:41.930096 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" podUID="f76b9e4b-12b0-438f-8952-4619872de7d1" containerName="payload-processing" containerID="cri-o://c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e" gracePeriod=25 Oct 30 19:12:42.258662 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.258626 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:12:42.294703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.294667 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn"] Oct 30 19:12:42.296027 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.296000 5959 cpu_manager.go:414] "containerMap: removing container" podUID="f76b9e4b-12b0-438f-8952-4619872de7d1" containerName="payload-processing" Oct 30 19:12:42.296151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.296030 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76b9e4b-12b0-438f-8952-4619872de7d1" containerName="payload-processing" Oct 30 19:12:42.296279 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.296263 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76b9e4b-12b0-438f-8952-4619872de7d1" containerName="payload-processing" Oct 30 19:12:42.304933 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.304912 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:42.311558 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.311002 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ecc266\"" Oct 30 19:12:42.311799 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.310407 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn"] Oct 30 19:12:42.313025 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.312975 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ecc266-dockercfg-4zksm\"" Oct 30 19:12:42.371514 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.371478 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-routing-overlay-volume\") pod \"f76b9e4b-12b0-438f-8952-4619872de7d1\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " Oct 30 19:12:42.371754 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.371528 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-plugins-config-volume\") pod \"f76b9e4b-12b0-438f-8952-4619872de7d1\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " Oct 30 19:12:42.371838 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.371764 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-provider-credentials\") pod \"f76b9e4b-12b0-438f-8952-4619872de7d1\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " Oct 30 19:12:42.371838 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.371821 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-kube-api-access-wbpbc\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-kube-api-access-wbpbc\") pod \"f76b9e4b-12b0-438f-8952-4619872de7d1\" (UID: \"f76b9e4b-12b0-438f-8952-4619872de7d1\") " Oct 30 19:12:42.372060 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.372022 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-routing-overlay-volume" pod "f76b9e4b-12b0-438f-8952-4619872de7d1" (UID: "f76b9e4b-12b0-438f-8952-4619872de7d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:42.372060 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.372038 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-plugins-config-volume" pod "f76b9e4b-12b0-438f-8952-4619872de7d1" (UID: "f76b9e4b-12b0-438f-8952-4619872de7d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:42.372724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.372704 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:42.372724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.372723 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b9e4b-12b0-438f-8952-4619872de7d1-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:42.374200 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.374174 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-kube-api-access-wbpbc" pod "f76b9e4b-12b0-438f-8952-4619872de7d1" (UID: "f76b9e4b-12b0-438f-8952-4619872de7d1"). InnerVolumeSpecName "kube-api-access-wbpbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:42.374382 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.374358 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-provider-credentials" pod "f76b9e4b-12b0-438f-8952-4619872de7d1" (UID: "f76b9e4b-12b0-438f-8952-4619872de7d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:42.474698 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.474573 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqxh\" (UniqueName: \"kubernetes.io/projected/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-kube-api-access-mvqxh\") pod \"payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn\" (UID: \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:42.474931 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.474913 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn\" (UID: \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:42.475334 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.475308 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:42.475458 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.475338 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbpbc\" (UniqueName: \"kubernetes.io/projected/f76b9e4b-12b0-438f-8952-4619872de7d1-kube-api-access-wbpbc\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:42.485458 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.485425 5959 generic.go:356] "Generic (PLEG): container finished" podID="f76b9e4b-12b0-438f-8952-4619872de7d1" containerID="c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e" exitCode=0 Oct 30 19:12:42.485596 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.485483 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" Oct 30 19:12:42.485596 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.485519 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" event={"ID":"f76b9e4b-12b0-438f-8952-4619872de7d1","Type":"ContainerDied","Data":"c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e"} Oct 30 19:12:42.485596 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.485587 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz" event={"ID":"f76b9e4b-12b0-438f-8952-4619872de7d1","Type":"ContainerDied","Data":"aacb6e3f615265feba28fdf3818e53fd83d49617c389862c53a18d1a3cc82285"} Oct 30 19:12:42.485804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.485669 5959 scope.go:122] "RemoveContainer" containerID="c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e" Oct 30 19:12:42.504822 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.504796 5959 scope.go:122] "RemoveContainer" containerID="c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e" Oct 30 19:12:42.505945 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:42.505913 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e\": container with ID starting with c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e not found: ID does not exist" containerID="c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e" Oct 30 19:12:42.506060 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.505954 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e"} err="failed to get container status \"c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e\": rpc error: code = NotFound desc = could not find container \"c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e\": container with ID starting with c560a0cc3a3f7dd5b464fcf8c6b45a81af4d97a08817b13e6607319e7005d93e not found: ID does not exist" Oct 30 19:12:42.515370 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.515341 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz"] Oct 30 19:12:42.519915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.519893 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e67d95/payload-processing-external-model-e2e-extmod-b-e67d95-5f8dw5tsz"] Oct 30 19:12:42.577497 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.577472 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn\" (UID: \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:42.577726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.577706 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqxh\" (UniqueName: \"kubernetes.io/projected/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-kube-api-access-mvqxh\") pod \"payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn\" (UID: \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:42.578275 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.578259 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn\" (UID: \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:42.586573 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.586551 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqxh\" (UniqueName: \"kubernetes.io/projected/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-kube-api-access-mvqxh\") pod \"payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn\" (UID: \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:42.633738 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.633712 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:42.932968 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:12:42.932941 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebdb22e_e42f_4ec2_be24_ab1e9b0cff73.slice/crio-2862e091930e909669b0b64d0f6b5de002b0919dde7d4ca3fc1f4e3896206e50 WatchSource:0}: Error finding container 2862e091930e909669b0b64d0f6b5de002b0919dde7d4ca3fc1f4e3896206e50: Status 404 returned error can't find the container with id 2862e091930e909669b0b64d0f6b5de002b0919dde7d4ca3fc1f4e3896206e50 Oct 30 19:12:42.935013 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:42.934986 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn"] Oct 30 19:12:43.495919 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:43.495879 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" event={"ID":"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73","Type":"ContainerStarted","Data":"bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0"} Oct 30 19:12:43.496094 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:43.495935 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" event={"ID":"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73","Type":"ContainerStarted","Data":"2862e091930e909669b0b64d0f6b5de002b0919dde7d4ca3fc1f4e3896206e50"} Oct 30 19:12:43.496170 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:43.496153 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:43.518663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:43.518582 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" podStartSLOduration=123.518564549 podStartE2EDuration="2m3.518564549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:43.517255395 +0000 UTC m=+2143.845666780" watchObservedRunningTime="2026-10-30 19:12:43.518564549 +0000 UTC m=+2143.846975935" Oct 30 19:12:43.801978 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:43.801894 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76b9e4b-12b0-438f-8952-4619872de7d1" path="/var/lib/kubelet/pods/f76b9e4b-12b0-438f-8952-4619872de7d1/volumes" Oct 30 19:12:44.430864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.430828 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" podUID="5fab75b5-9db9-415e-86be-7ea8f7d5e205" containerName="payload-processing" containerID="cri-o://6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd" gracePeriod=25 Oct 30 19:12:44.757079 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.757049 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:44.799592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.799554 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h"] Oct 30 19:12:44.800981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.800953 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5fab75b5-9db9-415e-86be-7ea8f7d5e205" containerName="payload-processing" Oct 30 19:12:44.800981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.800980 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fab75b5-9db9-415e-86be-7ea8f7d5e205" containerName="payload-processing" Oct 30 19:12:44.801211 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.801194 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fab75b5-9db9-415e-86be-7ea8f7d5e205" containerName="payload-processing" Oct 30 19:12:44.822198 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.822166 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h"] Oct 30 19:12:44.822388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.822287 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:44.827496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.825848 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-routing-overlay-volume\") pod \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " Oct 30 19:12:44.827496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.825965 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-kube-api-access-k4knn\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-kube-api-access-k4knn\") pod \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " Oct 30 19:12:44.827496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.826298 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-routing-overlay-volume" pod "5fab75b5-9db9-415e-86be-7ea8f7d5e205" (UID: "5fab75b5-9db9-415e-86be-7ea8f7d5e205"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:44.827496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.826379 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-plugins-config-volume\") pod \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " Oct 30 19:12:44.827496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.826417 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-provider-credentials\") pod \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\" (UID: \"5fab75b5-9db9-415e-86be-7ea8f7d5e205\") " Oct 30 19:12:44.827496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.827329 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:44.828345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.828322 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-plugins-config-volume" pod "5fab75b5-9db9-415e-86be-7ea8f7d5e205" (UID: "5fab75b5-9db9-415e-86be-7ea8f7d5e205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:44.840430 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.830110 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-provider-credentials" pod "5fab75b5-9db9-415e-86be-7ea8f7d5e205" (UID: "5fab75b5-9db9-415e-86be-7ea8f7d5e205"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:44.840430 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.830219 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-kube-api-access-k4knn" pod "5fab75b5-9db9-415e-86be-7ea8f7d5e205" (UID: "5fab75b5-9db9-415e-86be-7ea8f7d5e205"). InnerVolumeSpecName "kube-api-access-k4knn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:44.975727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.948835 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:44.975727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.948938 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:44.975727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.948974 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4st4q\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-kube-api-access-4st4q\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:44.975727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.949329 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:44.975727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.949551 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab75b5-9db9-415e-86be-7ea8f7d5e205-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:44.975727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.949570 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:44.975727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:44.949585 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4knn\" (UniqueName: \"kubernetes.io/projected/5fab75b5-9db9-415e-86be-7ea8f7d5e205-kube-api-access-k4knn\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:45.054631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.054210 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.054631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.054423 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.054631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.054509 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.054631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.054538 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4st4q\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-kube-api-access-4st4q\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.061633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.056470 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.065429 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.064908 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.109092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.108995 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.125636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.119273 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4st4q\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-kube-api-access-4st4q\") pod \"payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.176998 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.176959 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:45.519274 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.519167 5959 generic.go:356] "Generic (PLEG): container finished" podID="5fab75b5-9db9-415e-86be-7ea8f7d5e205" containerID="6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd" exitCode=0 Oct 30 19:12:45.519274 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.519255 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" event={"ID":"5fab75b5-9db9-415e-86be-7ea8f7d5e205","Type":"ContainerDied","Data":"6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd"} Oct 30 19:12:45.519943 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.519272 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" Oct 30 19:12:45.519943 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.519309 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r" event={"ID":"5fab75b5-9db9-415e-86be-7ea8f7d5e205","Type":"ContainerDied","Data":"4270abfd6534dd4120f4e794f3abe9ffd9c915532c6403d25c722f130a1b4157"} Oct 30 19:12:45.519943 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.519338 5959 scope.go:122] "RemoveContainer" containerID="6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd" Oct 30 19:12:45.533585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.533556 5959 scope.go:122] "RemoveContainer" containerID="6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd" Oct 30 19:12:45.533915 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:45.533882 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd\": container with ID starting with 6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd not found: ID does not exist" containerID="6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd" Oct 30 19:12:45.534010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.533914 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd"} err="failed to get container status \"6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd\": rpc error: code = NotFound desc = could not find container \"6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd\": container with ID starting with 6fc54ab4f97b90bf7f13753047b703325dc2442a28a151d6adb65d19a8c937dd not found: ID does not exist" Oct 30 19:12:45.547335 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.547309 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r"] Oct 30 19:12:45.551546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.551522 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-78dbgpd8r"] Oct 30 19:12:45.596727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.596697 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h"] Oct 30 19:12:45.597588 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:12:45.597561 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241fa83e_edc6_4002_99fa_b70f6fd0cd87.slice/crio-8a827898e6541d70470d88a460e766acb8109371d8f1b0f4f4772a86f2b70487 WatchSource:0}: Error finding container 8a827898e6541d70470d88a460e766acb8109371d8f1b0f4f4772a86f2b70487: Status 404 returned error can't find the container with id 8a827898e6541d70470d88a460e766acb8109371d8f1b0f4f4772a86f2b70487 Oct 30 19:12:45.883734 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:45.882686 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fab75b5-9db9-415e-86be-7ea8f7d5e205" path="/var/lib/kubelet/pods/5fab75b5-9db9-415e-86be-7ea8f7d5e205/volumes" Oct 30 19:12:46.544127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:46.544089 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" event={"ID":"241fa83e-edc6-4002-99fa-b70f6fd0cd87","Type":"ContainerStarted","Data":"ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76"} Oct 30 19:12:46.544582 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:46.544138 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" event={"ID":"241fa83e-edc6-4002-99fa-b70f6fd0cd87","Type":"ContainerStarted","Data":"8a827898e6541d70470d88a460e766acb8109371d8f1b0f4f4772a86f2b70487"} Oct 30 19:12:46.544582 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:46.544240 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:46.564764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:46.564707 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" podStartSLOduration=37.56468704 podStartE2EDuration="37.56468704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:46.562888103 +0000 UTC m=+2146.891299514" watchObservedRunningTime="2026-10-30 19:12:46.56468704 +0000 UTC m=+2146.893098427" Oct 30 19:12:54.122769 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:54.122732 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx"] Oct 30 19:12:54.129489 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:54.129455 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h"] Oct 30 19:12:54.134508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:54.134483 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:54.516512 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:54.515345 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:12:57.783526 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:57.783485 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:12:59.130017 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.129941 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" podUID="241fa83e-edc6-4002-99fa-b70f6fd0cd87" containerName="payload-processing" containerID="cri-o://ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76" gracePeriod=25 Oct 30 19:12:59.130661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.130202 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" podUID="2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" containerName="payload-processing" containerID="cri-o://ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252" gracePeriod=25 Oct 30 19:12:59.473597 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.473567 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:59.482798 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.482771 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:12:59.503091 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.503056 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9"] Oct 30 19:12:59.504069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.504047 5959 cpu_manager.go:414] "containerMap: removing container" podUID="241fa83e-edc6-4002-99fa-b70f6fd0cd87" containerName="payload-processing" Oct 30 19:12:59.504069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.504067 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241fa83e-edc6-4002-99fa-b70f6fd0cd87" containerName="payload-processing" Oct 30 19:12:59.504235 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.504109 5959 cpu_manager.go:414] "containerMap: removing container" podUID="2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" containerName="payload-processing" Oct 30 19:12:59.504235 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.504114 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" containerName="payload-processing" Oct 30 19:12:59.504327 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.504242 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="241fa83e-edc6-4002-99fa-b70f6fd0cd87" containerName="payload-processing" Oct 30 19:12:59.504327 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.504259 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" containerName="payload-processing" Oct 30 19:12:59.508113 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.508093 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.514967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.514910 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"agc-cred-fcfd33bd0bc0\"" Oct 30 19:12:59.514967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.514928 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e51c10\"" Oct 30 19:12:59.514967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.514928 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"openshift-service-ca.crt\"" Oct 30 19:12:59.515234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.514988 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"routing-overlay\"" Oct 30 19:12:59.515528 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.515506 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"payload-processing-external-model-e2e--24cd22ea-dockercfg-7db2g\"" Oct 30 19:12:59.515759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.515707 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"kube-root-ca.crt\"" Oct 30 19:12:59.516878 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.516858 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9"] Oct 30 19:12:59.530668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.529411 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-plugins-config-volume\") pod \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " Oct 30 19:12:59.530668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.529513 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-kube-api-access-4st4q\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-kube-api-access-4st4q\") pod \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " Oct 30 19:12:59.530668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.529729 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-routing-overlay-volume\") pod \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " Oct 30 19:12:59.530668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.529854 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-kube-api-access-dgq29\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-kube-api-access-dgq29\") pod \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " Oct 30 19:12:59.530668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.530073 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-routing-overlay-volume\") pod \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " Oct 30 19:12:59.530668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.530201 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials\") pod \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " Oct 30 19:12:59.530668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.530553 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-provider-credentials\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-provider-credentials\") pod \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\" (UID: \"241fa83e-edc6-4002-99fa-b70f6fd0cd87\") " Oct 30 19:12:59.539388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.533081 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4"] Oct 30 19:12:59.542006 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.539485 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-plugins-config-volume\") pod \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\" (UID: \"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428\") " Oct 30 19:12:59.542006 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.540628 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.542006 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.540718 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.542006 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.540904 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.542006 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.540993 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcqq\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-kube-api-access-kwcqq\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.549425 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.549395 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-routing-overlay-volume" pod "241fa83e-edc6-4002-99fa-b70f6fd0cd87" (UID: "241fa83e-edc6-4002-99fa-b70f6fd0cd87"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:59.549577 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.549480 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-plugins-config-volume" pod "241fa83e-edc6-4002-99fa-b70f6fd0cd87" (UID: "241fa83e-edc6-4002-99fa-b70f6fd0cd87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:59.551803 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.551784 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-kube-api-access-4st4q" pod "241fa83e-edc6-4002-99fa-b70f6fd0cd87" (UID: "241fa83e-edc6-4002-99fa-b70f6fd0cd87"). InnerVolumeSpecName "kube-api-access-4st4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:59.552077 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.552044 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-provider-credentials" pod "241fa83e-edc6-4002-99fa-b70f6fd0cd87" (UID: "241fa83e-edc6-4002-99fa-b70f6fd0cd87"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:59.552510 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.552432 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-plugins-config-volume" pod "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:59.553872 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.553842 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-routing-overlay-volume" pod "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:59.557684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.557587 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials" pod "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:59.558125 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.558098 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-kube-api-access-dgq29" pod "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" (UID: "2a2eeb47-ef9d-4ab2-9d92-71d5045e7428"). InnerVolumeSpecName "kube-api-access-dgq29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:59.563729 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.563704 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.565941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.565916 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4"] Oct 30 19:12:59.616018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.615986 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"e2e-cfl-5b9cb6-provider-cred\"" Oct 30 19:12:59.643099 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643075 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.643245 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643122 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcqq\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-kube-api-access-kwcqq\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.643245 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643172 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.643245 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643231 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.643472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643442 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-kube-api-access-mb66c\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.643577 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643560 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.643664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643596 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.643723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643669 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.643778 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:59.643714 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e51c10/agc-cred-fcfd33bd0bc0: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:12:59.643778 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:59.643736 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:12:59.643870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643797 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.643870 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:59.643816 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials podName:b5616cec-01b6-4dbd-b1be-918be57e8075 nodeName:}" failed. No retries permitted until 2026-10-30 19:13:00.143795349 +0000 UTC m=+2160.472206727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" (UID: "b5616cec-01b6-4dbd-b1be-918be57e8075") : secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:12:59.643870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643860 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:59.643969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643880 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:59.643969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643900 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:59.643969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643915 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4st4q\" (UniqueName: \"kubernetes.io/projected/241fa83e-edc6-4002-99fa-b70f6fd0cd87-kube-api-access-4st4q\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:59.643969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643931 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/241fa83e-edc6-4002-99fa-b70f6fd0cd87-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:59.643969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643945 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgq29\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-kube-api-access-dgq29\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:59.643969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643958 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:59.643969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.643972 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:59.644346 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.644327 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.654202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.654181 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcqq\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-kube-api-access-kwcqq\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:12:59.745022 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.744947 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-kube-api-access-mb66c\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.745169 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.745037 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.745211 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.745165 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.745211 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.745203 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.745731 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.745707 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.745901 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.745857 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.747813 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.747790 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.753089 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.753068 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-kube-api-access-mb66c\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:12:59.778766 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.778738 5959 generic.go:356] "Generic (PLEG): container finished" podID="241fa83e-edc6-4002-99fa-b70f6fd0cd87" containerID="ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76" exitCode=0 Oct 30 19:12:59.778909 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.778794 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" Oct 30 19:12:59.778909 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.778832 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" event={"ID":"241fa83e-edc6-4002-99fa-b70f6fd0cd87","Type":"ContainerDied","Data":"ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76"} Oct 30 19:12:59.778909 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.778882 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h" event={"ID":"241fa83e-edc6-4002-99fa-b70f6fd0cd87","Type":"ContainerDied","Data":"8a827898e6541d70470d88a460e766acb8109371d8f1b0f4f4772a86f2b70487"} Oct 30 19:12:59.779098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.778911 5959 scope.go:122] "RemoveContainer" containerID="ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76" Oct 30 19:12:59.780643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.780617 5959 generic.go:356] "Generic (PLEG): container finished" podID="2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" containerID="ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252" exitCode=0 Oct 30 19:12:59.780755 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.780629 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" event={"ID":"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428","Type":"ContainerDied","Data":"ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252"} Oct 30 19:12:59.780755 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.780708 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" Oct 30 19:12:59.780877 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.780753 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx" event={"ID":"2a2eeb47-ef9d-4ab2-9d92-71d5045e7428","Type":"ContainerDied","Data":"20f31fcef6d95b5688076ce2218bcdbc918a97f0e9e27161933752be57a051d6"} Oct 30 19:12:59.792653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.792634 5959 scope.go:122] "RemoveContainer" containerID="ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76" Oct 30 19:12:59.793009 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:59.792988 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76\": container with ID starting with ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76 not found: ID does not exist" containerID="ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76" Oct 30 19:12:59.793102 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.793016 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76"} err="failed to get container status \"ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76\": rpc error: code = NotFound desc = could not find container \"ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76\": container with ID starting with ef8052f570d1c4a82b60cda24893ace2f6041056cac4782caae35271e56d7e76 not found: ID does not exist" Oct 30 19:12:59.793102 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.793033 5959 scope.go:122] "RemoveContainer" containerID="ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252" Oct 30 19:12:59.804880 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.804852 5959 scope.go:122] "RemoveContainer" containerID="ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252" Oct 30 19:12:59.805222 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:12:59.805199 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252\": container with ID starting with ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252 not found: ID does not exist" containerID="ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252" Oct 30 19:12:59.805343 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.805230 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252"} err="failed to get container status \"ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252\": rpc error: code = NotFound desc = could not find container \"ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252\": container with ID starting with ab86247bcdba5d35c3cf0b7b73a0377ddfe4da95d68e04a834d8d783fc010252 not found: ID does not exist" Oct 30 19:12:59.830797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.830766 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h"] Oct 30 19:12:59.844092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.844060 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-6b9d7sv7h"] Oct 30 19:12:59.850783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.850753 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx"] Oct 30 19:12:59.857459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.857392 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4a477d/payload-processing-external-model-e2e-extmod-b-4a477d-5bf6wmkjx"] Oct 30 19:12:59.883709 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.883677 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"payload-processing-external-model-e2e--24cd22ea-dockercfg-7db2g\"" Oct 30 19:12:59.891382 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:12:59.891356 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:13:00.153488 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:00.153451 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:13:00.153919 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:00.153595 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e51c10/agc-cred-fcfd33bd0bc0: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:00.153919 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:00.153629 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:00.153919 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:00.153701 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials podName:b5616cec-01b6-4dbd-b1be-918be57e8075 nodeName:}" failed. No retries permitted until 2026-10-30 19:13:01.153681395 +0000 UTC m=+2161.482092777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" (UID: "b5616cec-01b6-4dbd-b1be-918be57e8075") : secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:00.191772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:00.191743 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4"] Oct 30 19:13:00.192773 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:13:00.192748 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5c26a7_8604_42f9_9372_b887480f173d.slice/crio-c9af29e26b57fd4151821bc7b478bd39866dc7c3dbc79b8bd6703f9e08fbf314 WatchSource:0}: Error finding container c9af29e26b57fd4151821bc7b478bd39866dc7c3dbc79b8bd6703f9e08fbf314: Status 404 returned error can't find the container with id c9af29e26b57fd4151821bc7b478bd39866dc7c3dbc79b8bd6703f9e08fbf314 Oct 30 19:13:00.794347 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:00.794309 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" event={"ID":"bd5c26a7-8604-42f9-9372-b887480f173d","Type":"ContainerStarted","Data":"48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1"} Oct 30 19:13:00.794562 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:00.794378 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:13:00.794562 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:00.794407 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" event={"ID":"bd5c26a7-8604-42f9-9372-b887480f173d","Type":"ContainerStarted","Data":"c9af29e26b57fd4151821bc7b478bd39866dc7c3dbc79b8bd6703f9e08fbf314"} Oct 30 19:13:00.832064 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:00.832015 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" podStartSLOduration=13.831998214 podStartE2EDuration="13.831998214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:00.827022591 +0000 UTC m=+2161.155433976" watchObservedRunningTime="2026-10-30 19:13:00.831998214 +0000 UTC m=+2161.160409598" Oct 30 19:13:01.187942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:01.187910 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:13:01.188343 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:01.188069 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e51c10/agc-cred-fcfd33bd0bc0: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:01.188343 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:01.188086 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:01.188343 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:01.188153 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials podName:b5616cec-01b6-4dbd-b1be-918be57e8075 nodeName:}" failed. No retries permitted until 2026-10-30 19:13:03.188134767 +0000 UTC m=+2163.516546152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" (UID: "b5616cec-01b6-4dbd-b1be-918be57e8075") : secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:01.800100 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:01.800067 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241fa83e-edc6-4002-99fa-b70f6fd0cd87" path="/var/lib/kubelet/pods/241fa83e-edc6-4002-99fa-b70f6fd0cd87/volumes" Oct 30 19:13:01.800552 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:01.800533 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2eeb47-ef9d-4ab2-9d92-71d5045e7428" path="/var/lib/kubelet/pods/2a2eeb47-ef9d-4ab2-9d92-71d5045e7428/volumes" Oct 30 19:13:03.238627 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:03.238579 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:13:03.239067 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:03.238731 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e51c10/agc-cred-fcfd33bd0bc0: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:03.239067 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:03.238750 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:03.239067 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:03.238803 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials podName:b5616cec-01b6-4dbd-b1be-918be57e8075 nodeName:}" failed. No retries permitted until 2026-10-30 19:13:07.238789116 +0000 UTC m=+2167.567200484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" (UID: "b5616cec-01b6-4dbd-b1be-918be57e8075") : secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:07.329536 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:07.329494 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:13:07.330010 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:07.329676 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e51c10/agc-cred-fcfd33bd0bc0: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:07.330010 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:07.329693 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:07.330010 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:07.329753 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials podName:b5616cec-01b6-4dbd-b1be-918be57e8075 nodeName:}" failed. No retries permitted until 2026-10-30 19:13:15.329737298 +0000 UTC m=+2175.658148673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" (UID: "b5616cec-01b6-4dbd-b1be-918be57e8075") : secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:13:07.883735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:07.883698 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:13:10.028545 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:10.022040 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:13:11.810540 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:11.810507 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:13:11.869905 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:11.869867 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9"] Oct 30 19:13:11.870548 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:11.870518 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" podUID="b5616cec-01b6-4dbd-b1be-918be57e8075" Oct 30 19:13:12.068837 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.068758 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:13:12.078683 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.078655 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:13:12.178965 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.178928 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-routing-overlay-volume\") pod \"b5616cec-01b6-4dbd-b1be-918be57e8075\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " Oct 30 19:13:12.179275 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.179244 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-kube-api-access-kwcqq\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-kube-api-access-kwcqq\") pod \"b5616cec-01b6-4dbd-b1be-918be57e8075\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " Oct 30 19:13:12.179366 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.179304 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-routing-overlay-volume" pod "b5616cec-01b6-4dbd-b1be-918be57e8075" (UID: "b5616cec-01b6-4dbd-b1be-918be57e8075"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:12.179366 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.179339 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-plugins-config-volume\") pod \"b5616cec-01b6-4dbd-b1be-918be57e8075\" (UID: \"b5616cec-01b6-4dbd-b1be-918be57e8075\") " Oct 30 19:13:12.179718 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.179698 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-plugins-config-volume" pod "b5616cec-01b6-4dbd-b1be-918be57e8075" (UID: "b5616cec-01b6-4dbd-b1be-918be57e8075"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:12.180038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.180022 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:12.180083 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.180042 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5616cec-01b6-4dbd-b1be-918be57e8075-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:12.181497 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.181477 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-kube-api-access-kwcqq" pod "b5616cec-01b6-4dbd-b1be-918be57e8075" (UID: "b5616cec-01b6-4dbd-b1be-918be57e8075"). InnerVolumeSpecName "kube-api-access-kwcqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:12.282186 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:12.282151 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcqq\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-kube-api-access-kwcqq\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:13.076760 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.076731 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9" Oct 30 19:13:13.110979 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.110944 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp"] Oct 30 19:13:13.117765 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.117734 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9"] Oct 30 19:13:13.117937 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.117840 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.122983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.122490 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6f956\"/\"routing-overlay\"" Oct 30 19:13:13.122983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.122579 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6f956\"/\"e2e-cmp-legacy-82f59a-provider-cred\"" Oct 30 19:13:13.122983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.122707 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6f956\"/\"payload-processing-external-model-e2e--8e3d1fad-dockercfg-qt46k\"" Oct 30 19:13:13.122983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.122790 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6f956\"/\"kube-root-ca.crt\"" Oct 30 19:13:13.122983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.122845 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6f956\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d6f956\"" Oct 30 19:13:13.122983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.122922 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6f956\"/\"openshift-service-ca.crt\"" Oct 30 19:13:13.122983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.122962 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8sqkr9"] Oct 30 19:13:13.128286 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.128260 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp"] Oct 30 19:13:13.204804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.204766 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wmx5\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-kube-api-access-6wmx5\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.205020 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.204886 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.205201 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.205181 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.205277 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.205217 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.205433 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.205410 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5616cec-01b6-4dbd-b1be-918be57e8075-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:13.307108 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.307065 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.307297 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.307116 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.307297 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.307247 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wmx5\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-kube-api-access-6wmx5\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.307401 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.307307 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.307906 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.307878 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.308063 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.308040 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.309993 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.309971 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.315589 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.315567 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wmx5\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-kube-api-access-6wmx5\") pod \"payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.439892 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.439853 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:13.729345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.729317 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp"] Oct 30 19:13:13.729903 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:13:13.729877 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94520350_11fd_4ef1_9cd6_cf86f644cb74.slice/crio-a179e050f963a1cb27fa6cd8824ee23efde139b83a2c126602b4490e14b2a0e7 WatchSource:0}: Error finding container a179e050f963a1cb27fa6cd8824ee23efde139b83a2c126602b4490e14b2a0e7: Status 404 returned error can't find the container with id a179e050f963a1cb27fa6cd8824ee23efde139b83a2c126602b4490e14b2a0e7 Oct 30 19:13:13.801932 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:13.801900 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5616cec-01b6-4dbd-b1be-918be57e8075" path="/var/lib/kubelet/pods/b5616cec-01b6-4dbd-b1be-918be57e8075/volumes" Oct 30 19:13:14.088290 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:14.088202 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" event={"ID":"94520350-11fd-4ef1-9cd6-cf86f644cb74","Type":"ContainerStarted","Data":"dcabfb551091b8029e048f69ddf0333e0f26bf99f66ffa58c198b1662e584bee"} Oct 30 19:13:14.088290 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:14.088268 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:14.088290 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:14.088290 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" event={"ID":"94520350-11fd-4ef1-9cd6-cf86f644cb74","Type":"ContainerStarted","Data":"a179e050f963a1cb27fa6cd8824ee23efde139b83a2c126602b4490e14b2a0e7"} Oct 30 19:13:14.108050 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:14.107998 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" podStartSLOduration=4.107985727 podStartE2EDuration="4.107985727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:14.106150152 +0000 UTC m=+2174.434561540" watchObservedRunningTime="2026-10-30 19:13:14.107985727 +0000 UTC m=+2174.436397112" Oct 30 19:13:22.466006 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:22.465971 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq"] Oct 30 19:13:22.466423 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:22.466222 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" containerID="cri-o://1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58" gracePeriod=30 Oct 30 19:13:24.082730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:24.082701 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv"] Oct 30 19:13:24.083170 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:24.082936 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" containerID="cri-o://ae38a8ef0b2fcbbe5d6a42c951dcceee95401ca8e6339b5ba4f425d8cbc601fc" gracePeriod=30 Oct 30 19:13:24.775273 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:24.775239 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4"] Oct 30 19:13:24.775522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:24.775497 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" containerID="cri-o://7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb" gracePeriod=30 Oct 30 19:13:25.099405 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:25.099328 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:13:27.764043 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.764015 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:13:27.789236 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:27.789197 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:13:27.808327 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.808299 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b"] Oct 30 19:13:27.817770 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.817188 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" Oct 30 19:13:27.818017 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.817988 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" Oct 30 19:13:27.818564 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.818530 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerName="istio-proxy" Oct 30 19:13:27.823452 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.823428 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b"] Oct 30 19:13:27.823584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.823555 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.827594 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.827574 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1dd09d-openshift-default-dockercfg-cj522\"" Oct 30 19:13:27.899297 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899253 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-socket\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899297 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899289 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-podinfo\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899563 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899458 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-certs\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899563 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899507 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-data\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899563 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899563 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-kube-api-access-s6fpb\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-kube-api-access-s6fpb\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899617 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istiod-ca-cert\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899598 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-socket" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:27.899743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899697 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-credential-socket\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899884 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899744 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-envoy\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899884 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899809 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-token\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-token\") pod \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\" (UID: \"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408\") " Oct 30 19:13:27.899884 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899807 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-certs" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:27.899884 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.899864 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-data" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:27.900068 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.900028 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-credential-socket" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:27.900158 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.900125 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istiod-ca-cert" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:27.900537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.900508 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.900685 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.900658 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.900930 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.900795 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.900930 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.900867 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.901056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.900942 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.901056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.900980 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.901056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.901034 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.901250 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.901215 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.901337 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.901317 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2fz\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-kube-api-access-8l2fz\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:27.901554 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.901526 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:27.901701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.901595 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:27.901701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.901654 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:27.901701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.901671 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:27.901701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.901683 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:27.902073 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.902051 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-podinfo" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:27.902357 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.902335 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-kube-api-access-s6fpb" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "kube-api-access-s6fpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:27.902357 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.902338 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-envoy" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:27.902486 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:27.902446 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-token" pod "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" (UID: "5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:28.005140 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.005089 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.005397 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.005370 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.005525 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.005439 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.005584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.005567 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.005740 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.005722 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2fz\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-kube-api-access-8l2fz\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.005928 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.005911 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.005981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.005957 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.006087 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.006069 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.006217 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.006199 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.006300 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.006285 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.006497 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.006470 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.006682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.006534 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:28.006877 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.006789 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:28.006877 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.006821 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:28.006877 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.006841 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6fpb\" (UniqueName: \"kubernetes.io/projected/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408-kube-api-access-s6fpb\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:28.007120 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.007009 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.007226 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.007203 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.007481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.007460 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.008851 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.008829 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.009657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.009592 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.016646 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.015866 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.020278 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.020241 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2fz\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-kube-api-access-8l2fz\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.141852 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.141816 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:28.228726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.228215 5959 generic.go:356] "Generic (PLEG): container finished" podID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" containerID="1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58" exitCode=0 Oct 30 19:13:28.228726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.228274 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" event={"ID":"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408","Type":"ContainerDied","Data":"1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58"} Oct 30 19:13:28.228726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.228314 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" event={"ID":"5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408","Type":"ContainerDied","Data":"ad47bee22de2ce5875fc0c008970fbc1d9304bb83400d0a5d4945e604437fa82"} Oct 30 19:13:28.228726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.228334 5959 scope.go:122] "RemoveContainer" containerID="1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58" Oct 30 19:13:28.228726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.228465 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq" Oct 30 19:13:28.245344 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.245320 5959 scope.go:122] "RemoveContainer" containerID="1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58" Oct 30 19:13:28.245760 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:28.245741 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58\": container with ID starting with 1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58 not found: ID does not exist" containerID="1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58" Oct 30 19:13:28.245803 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.245771 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58"} err="failed to get container status \"1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58\": rpc error: code = NotFound desc = could not find container \"1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58\": container with ID starting with 1f4c032bffdedafb700ddf491f83ed5d9b85de24fba72780003a6e6f8fda7b58 not found: ID does not exist" Oct 30 19:13:28.259631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.259538 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq"] Oct 30 19:13:28.267931 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.267838 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-d6cfq"] Oct 30 19:13:28.432334 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:13:28.432302 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e262f71_a759_46c9_8bc6_99f47b2fbc63.slice/crio-5e0f4e10f144d26382dc00db9f6ea6a0d0db7e621c529cfa4c82311180b3f73a WatchSource:0}: Error finding container 5e0f4e10f144d26382dc00db9f6ea6a0d0db7e621c529cfa4c82311180b3f73a: Status 404 returned error can't find the container with id 5e0f4e10f144d26382dc00db9f6ea6a0d0db7e621c529cfa4c82311180b3f73a Oct 30 19:13:28.433083 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.433060 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b"] Oct 30 19:13:28.435887 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.435856 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:28.435995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.435934 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:28.435995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:28.435977 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:29.063362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.063329 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658"] Oct 30 19:13:29.244287 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.244251 5959 generic.go:356] "Generic (PLEG): container finished" podID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerID="ae38a8ef0b2fcbbe5d6a42c951dcceee95401ca8e6339b5ba4f425d8cbc601fc" exitCode=0 Oct 30 19:13:29.244457 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.244370 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" event={"ID":"61e75b5f-85d1-4d91-8b25-2890951b57f8","Type":"ContainerDied","Data":"ae38a8ef0b2fcbbe5d6a42c951dcceee95401ca8e6339b5ba4f425d8cbc601fc"} Oct 30 19:13:29.246270 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.246247 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" event={"ID":"2e262f71-a759-46c9-8bc6-99f47b2fbc63","Type":"ContainerStarted","Data":"d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956"} Oct 30 19:13:29.246351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.246291 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" event={"ID":"2e262f71-a759-46c9-8bc6-99f47b2fbc63","Type":"ContainerStarted","Data":"5e0f4e10f144d26382dc00db9f6ea6a0d0db7e621c529cfa4c82311180b3f73a"} Oct 30 19:13:29.273647 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.271908 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" podStartSLOduration=70.271892301 podStartE2EDuration="1m10.271892301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:29.267567094 +0000 UTC m=+2189.595978482" watchObservedRunningTime="2026-10-30 19:13:29.271892301 +0000 UTC m=+2189.600303686" Oct 30 19:13:29.554723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.551121 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:13:29.583114 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583022 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-certs\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.583287 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583227 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-token\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-token\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.583364 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583315 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-kube-api-access-gt7sm\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-kube-api-access-gt7sm\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.583577 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583554 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e75b5f-85d1-4d91-8b25-2890951b57f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e75b5f-85d1-4d91-8b25-2890951b57f8-istiod-ca-cert\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.583723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583627 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-credential-socket\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.583723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583654 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-socket\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.583723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583713 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-podinfo\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.583893 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583771 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-data\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.583893 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.583808 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-envoy\") pod \"61e75b5f-85d1-4d91-8b25-2890951b57f8\" (UID: \"61e75b5f-85d1-4d91-8b25-2890951b57f8\") " Oct 30 19:13:29.585743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.585699 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e75b5f-85d1-4d91-8b25-2890951b57f8-istiod-ca-cert" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:29.586018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.585995 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-certs" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.586224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.586202 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-data" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.586471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.586416 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-credential-socket" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.586886 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.586851 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-socket" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.589993 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.589956 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-podinfo" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:29.590097 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.590025 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-kube-api-access-gt7sm" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "kube-api-access-gt7sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:29.590474 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.590441 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-envoy" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.591902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.591870 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-token" pod "61e75b5f-85d1-4d91-8b25-2890951b57f8" (UID: "61e75b5f-85d1-4d91-8b25-2890951b57f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:29.598200 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.598180 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d"] Oct 30 19:13:29.607438 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.607410 5959 cpu_manager.go:414] "containerMap: removing container" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" Oct 30 19:13:29.607598 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.607585 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" Oct 30 19:13:29.608557 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.608010 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" containerName="istio-proxy" Oct 30 19:13:29.625706 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.622891 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d"] Oct 30 19:13:29.625706 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.623024 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.694660 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.691740 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ae6370-0587-43e9-88d0-96594764269f-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.699089 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.695037 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.699089 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.695104 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.699804 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkdxz\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-kube-api-access-gkdxz\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.699906 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700007 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700136 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700171 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700197 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ae6370-0587-43e9-88d0-96594764269f-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700361 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e75b5f-85d1-4d91-8b25-2890951b57f8-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700377 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700434 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700454 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700468 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700481 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700495 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e75b5f-85d1-4d91-8b25-2890951b57f8-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700514 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.700670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.700528 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7sm\" (UniqueName: \"kubernetes.io/projected/61e75b5f-85d1-4d91-8b25-2890951b57f8-kube-api-access-gt7sm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.814810 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.814100 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.821704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.813667 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.822091 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.822065 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.822237 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.822220 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ae6370-0587-43e9-88d0-96594764269f-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.822454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.822436 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ae6370-0587-43e9-88d0-96594764269f-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.822672 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.822654 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.822818 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.822797 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.823232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.823204 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkdxz\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-kube-api-access-gkdxz\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.823351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.823284 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.823412 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.823349 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.824669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.824589 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.824778 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.824661 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.824906 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.824888 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.825985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.825341 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ae6370-0587-43e9-88d0-96594764269f-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.827029 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.827007 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.827312 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.827292 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408" path="/var/lib/kubelet/pods/5eaf6ddb-5c3f-43dc-b49f-d23e1bb17408/volumes" Oct 30 19:13:29.827576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.827556 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ae6370-0587-43e9-88d0-96594764269f-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.831367 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.831343 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.832246 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.832226 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkdxz\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-kube-api-access-gkdxz\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:29.966449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:29.966412 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:30.152840 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.152636 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:30.157259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.154726 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 19:13:30.157259 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.154779 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 19:13:30.244708 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.233190 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 19:13:30.279814 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.279110 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74"] Oct 30 19:13:30.281641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.281214 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" Oct 30 19:13:30.281641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.281240 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" Oct 30 19:13:30.281641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.281480 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerName="istio-proxy" Oct 30 19:13:30.282234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.282205 5959 generic.go:356] "Generic (PLEG): container finished" podID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" containerID="7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb" exitCode=0 Oct 30 19:13:30.282408 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.282388 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" Oct 30 19:13:30.284544 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.284526 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" Oct 30 19:13:30.285864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.285824 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" event={"ID":"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422","Type":"ContainerDied","Data":"7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb"} Oct 30 19:13:30.286026 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.286009 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4" event={"ID":"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422","Type":"ContainerDied","Data":"44ca3d14a5ba654dbf5d63e773f4ef321bdaee9ce89de84562e31cdcb4fa0987"} Oct 30 19:13:30.286127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.286115 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv" event={"ID":"61e75b5f-85d1-4d91-8b25-2890951b57f8","Type":"ContainerDied","Data":"c69c8e498cd139b76e8419abc2a8857fe15b514cced3a04d511f49d9549cd153"} Oct 30 19:13:30.286230 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.286217 5959 scope.go:122] "RemoveContainer" containerID="7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb" Oct 30 19:13:30.286389 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.286283 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.319633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.296851 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74"] Oct 30 19:13:30.336432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.336401 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-certs\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.336722 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.336701 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-data\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.336885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.336870 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-socket\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.337122 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.337107 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-envoy\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.337278 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.337261 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-kube-api-access-s5lts\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-kube-api-access-s5lts\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.337400 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.337387 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-credential-socket\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.337544 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.337529 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istiod-ca-cert\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.337799 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.337781 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-token\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-token\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.338036 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.338019 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-podinfo\") pod \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\" (UID: \"5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422\") " Oct 30 19:13:30.338696 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.338672 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.339132 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.339107 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzdxg\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-kube-api-access-tzdxg\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.339417 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.339394 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.339595 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.339580 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.339755 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.339739 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.339975 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.339960 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.340081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.340067 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97adfae8-ff7c-454b-b04b-72d64bb564b6-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.340205 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.340192 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.340287 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.337785 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-certs" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:30.340367 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.337954 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-data" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:30.340451 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.338148 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-credential-socket" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:30.340529 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.338687 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-socket" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:30.340597 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.338998 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istiod-ca-cert" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:30.340699 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.340349 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.340962 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.340932 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.341067 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.341054 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.341160 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.341147 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.341240 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.341229 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.341314 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.341304 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.343737 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.342820 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-kube-api-access-s5lts" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "kube-api-access-s5lts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:30.343737 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.343363 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-token" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:30.343737 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.343685 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-envoy" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:30.346373 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.346064 5959 scope.go:122] "RemoveContainer" containerID="7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb" Oct 30 19:13:30.346373 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.346194 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-podinfo" pod "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" (UID: "5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:30.346563 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:30.346542 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb\": container with ID starting with 7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb not found: ID does not exist" containerID="7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb" Oct 30 19:13:30.346709 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.346688 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb"} err="failed to get container status \"7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb\": rpc error: code = NotFound desc = could not find container \"7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb\": container with ID starting with 7c24fc53fbc5be02d5c082d274328d757dcab5f2c800fb6dc449db2cc3408ecb not found: ID does not exist" Oct 30 19:13:30.346795 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.346787 5959 scope.go:122] "RemoveContainer" containerID="ae38a8ef0b2fcbbe5d6a42c951dcceee95401ca8e6339b5ba4f425d8cbc601fc" Oct 30 19:13:30.363114 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.363044 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv"] Oct 30 19:13:30.378432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.378305 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-l5fqv"] Oct 30 19:13:30.444791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.444760 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.444791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.444795 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97adfae8-ff7c-454b-b04b-72d64bb564b6-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.444982 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.444817 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.444982 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.444850 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.444982 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.444906 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.445107 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445032 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdxg\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-kube-api-access-tzdxg\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.445107 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445089 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.445166 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445131 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.445166 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445152 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.445514 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445237 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.445821 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445763 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5lts\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-kube-api-access-s5lts\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.445942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445826 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.445942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445843 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:30.445942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445568 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.445942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445866 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.445942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.445781 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.446120 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.446074 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97adfae8-ff7c-454b-b04b-72d64bb564b6-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.446164 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.446145 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.447801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.447780 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.448233 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.448200 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.461477 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.461454 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.464789 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.464768 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzdxg\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-kube-api-access-tzdxg\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.529757 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.529718 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d"] Oct 30 19:13:30.533332 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:13:30.533126 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ae6370_0587_43e9_88d0_96594764269f.slice/crio-28b1a89c9480d1341233652674d621d7b5bc04f826387df30e9dd8b2334de4c9 WatchSource:0}: Error finding container 28b1a89c9480d1341233652674d621d7b5bc04f826387df30e9dd8b2334de4c9: Status 404 returned error can't find the container with id 28b1a89c9480d1341233652674d621d7b5bc04f826387df30e9dd8b2334de4c9 Oct 30 19:13:30.536480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.536450 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:30.536566 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.536520 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:30.536566 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.536551 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:30.639847 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.639820 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4"] Oct 30 19:13:30.640789 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.640758 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:30.646097 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.646072 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-bpjw4"] Oct 30 19:13:30.994007 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:30.993812 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74"] Oct 30 19:13:30.994655 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:13:30.994630 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97adfae8_ff7c_454b_b04b_72d64bb564b6.slice/crio-8deeb0a25e180027c994fe009cfdb23d99408cc2e3d316ca9218d001ac568b09 WatchSource:0}: Error finding container 8deeb0a25e180027c994fe009cfdb23d99408cc2e3d316ca9218d001ac568b09: Status 404 returned error can't find the container with id 8deeb0a25e180027c994fe009cfdb23d99408cc2e3d316ca9218d001ac568b09 Oct 30 19:13:31.026118 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.025859 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:31.026118 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.025945 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:31.026118 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.025992 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:13:31.143506 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.143472 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 19:13:31.143726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.143529 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 19:13:31.355278 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.355160 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" event={"ID":"97adfae8-ff7c-454b-b04b-72d64bb564b6","Type":"ContainerStarted","Data":"33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6"} Oct 30 19:13:31.355278 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.355249 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" event={"ID":"97adfae8-ff7c-454b-b04b-72d64bb564b6","Type":"ContainerStarted","Data":"8deeb0a25e180027c994fe009cfdb23d99408cc2e3d316ca9218d001ac568b09"} Oct 30 19:13:31.365008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.364975 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" event={"ID":"d9ae6370-0587-43e9-88d0-96594764269f","Type":"ContainerStarted","Data":"1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72"} Oct 30 19:13:31.365116 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.365034 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" event={"ID":"d9ae6370-0587-43e9-88d0-96594764269f","Type":"ContainerStarted","Data":"28b1a89c9480d1341233652674d621d7b5bc04f826387df30e9dd8b2334de4c9"} Oct 30 19:13:31.384068 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.384019 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" podStartSLOduration=72.384004379 podStartE2EDuration="1m12.384004379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:31.377729 +0000 UTC m=+2191.706140385" watchObservedRunningTime="2026-10-30 19:13:31.384004379 +0000 UTC m=+2191.712415766" Oct 30 19:13:31.425693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.425509 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podStartSLOduration=72.425490219 podStartE2EDuration="1m12.425490219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:31.424949869 +0000 UTC m=+2191.753361255" watchObservedRunningTime="2026-10-30 19:13:31.425490219 +0000 UTC m=+2191.753901605" Oct 30 19:13:31.642115 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.642072 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:31.836060 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.836025 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422" path="/var/lib/kubelet/pods/5cbb74a6-5c3e-4bf5-ae5b-b8fbcb708422/volumes" Oct 30 19:13:31.850435 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.850398 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e75b5f-85d1-4d91-8b25-2890951b57f8" path="/var/lib/kubelet/pods/61e75b5f-85d1-4d91-8b25-2890951b57f8/volumes" Oct 30 19:13:31.968442 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.968372 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:31.969727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.969699 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 19:13:31.969861 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:31.969749 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 19:13:32.150633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:32.150487 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 19:13:32.150633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:32.150549 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 19:13:32.771829 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:32.769841 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 19:13:32.771829 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:32.769914 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 19:13:32.984630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:32.967715 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 19:13:32.984630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:32.967782 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 19:13:33.147001 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:33.146146 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 19:13:33.147001 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:33.146210 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 19:13:33.651693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:33.645894 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 19:13:33.651693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:33.645948 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 19:13:33.991681 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:33.988333 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 19:13:33.991681 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:33.988393 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 19:13:34.068633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.064751 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" podUID="87e0e9ca-04c2-461d-9749-7a31da9847cb" containerName="payload-processing" containerID="cri-o://89e31ab7c0569ac81517e4be4100d9139f640cafd58ac84cecbc5fcdd81e52f7" gracePeriod=25 Oct 30 19:13:34.149874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.145961 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 19:13:34.149874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.146033 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 19:13:34.645119 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.642989 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 19:13:34.645119 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.643052 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 19:13:34.870940 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.870904 5959 generic.go:356] "Generic (PLEG): container finished" podID="87e0e9ca-04c2-461d-9749-7a31da9847cb" containerID="89e31ab7c0569ac81517e4be4100d9139f640cafd58ac84cecbc5fcdd81e52f7" exitCode=0 Oct 30 19:13:34.871355 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.871330 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" event={"ID":"87e0e9ca-04c2-461d-9749-7a31da9847cb","Type":"ContainerDied","Data":"89e31ab7c0569ac81517e4be4100d9139f640cafd58ac84cecbc5fcdd81e52f7"} Oct 30 19:13:34.976631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.972204 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:13:34.976631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.976092 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 19:13:34.976631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:34.976141 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 19:13:35.184596 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.184550 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-plugins-config-volume\") pod \"87e0e9ca-04c2-461d-9749-7a31da9847cb\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " Oct 30 19:13:35.185325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.185297 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-plugins-config-volume" pod "87e0e9ca-04c2-461d-9749-7a31da9847cb" (UID: "87e0e9ca-04c2-461d-9749-7a31da9847cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:35.185515 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.185493 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-routing-overlay-volume\") pod \"87e0e9ca-04c2-461d-9749-7a31da9847cb\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " Oct 30 19:13:35.185801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.185767 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-provider-credentials\") pod \"87e0e9ca-04c2-461d-9749-7a31da9847cb\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " Oct 30 19:13:35.186477 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.186111 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-routing-overlay-volume" pod "87e0e9ca-04c2-461d-9749-7a31da9847cb" (UID: "87e0e9ca-04c2-461d-9749-7a31da9847cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:35.186768 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.186742 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-kube-api-access-6cs4g\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-kube-api-access-6cs4g\") pod \"87e0e9ca-04c2-461d-9749-7a31da9847cb\" (UID: \"87e0e9ca-04c2-461d-9749-7a31da9847cb\") " Oct 30 19:13:35.188150 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.188114 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:35.188311 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.188297 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e0e9ca-04c2-461d-9749-7a31da9847cb-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:35.189037 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.189016 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-provider-credentials" pod "87e0e9ca-04c2-461d-9749-7a31da9847cb" (UID: "87e0e9ca-04c2-461d-9749-7a31da9847cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:35.192249 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.192227 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-kube-api-access-6cs4g" pod "87e0e9ca-04c2-461d-9749-7a31da9847cb" (UID: "87e0e9ca-04c2-461d-9749-7a31da9847cb"). InnerVolumeSpecName "kube-api-access-6cs4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:35.289946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.289896 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cs4g\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-kube-api-access-6cs4g\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:35.289946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.289924 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e0e9ca-04c2-461d-9749-7a31da9847cb-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:35.390636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.389956 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:35.395202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.394555 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:35.401638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.399856 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:13:35.650442 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.650068 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 19:13:35.650442 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.650127 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 19:13:35.881007 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.880975 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" event={"ID":"87e0e9ca-04c2-461d-9749-7a31da9847cb","Type":"ContainerDied","Data":"2c2f421de3f76a9fa7e7fd0bfcdeaa219e78a682f56b49ce50b96e467a1c2050"} Oct 30 19:13:35.881232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.881220 5959 scope.go:122] "RemoveContainer" containerID="89e31ab7c0569ac81517e4be4100d9139f640cafd58ac84cecbc5fcdd81e52f7" Oct 30 19:13:35.881386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.881016 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658" Oct 30 19:13:35.970385 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.970304 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658"] Oct 30 19:13:35.971300 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.970722 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 19:13:35.971300 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.970779 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 19:13:35.975504 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:35.975476 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-5cf64n658"] Oct 30 19:13:36.658501 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:36.649052 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 19:13:36.658501 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:36.649117 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 19:13:36.977726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:36.974723 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 19:13:36.977726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:36.974781 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 19:13:37.666016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:37.663231 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 19:13:37.666016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:37.663294 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 19:13:37.818292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:37.818252 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e0e9ca-04c2-461d-9749-7a31da9847cb" path="/var/lib/kubelet/pods/87e0e9ca-04c2-461d-9749-7a31da9847cb/volumes" Oct 30 19:13:37.967414 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:37.967319 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 19:13:37.967414 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:37.967384 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 19:13:38.728903 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:38.728869 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:38.729636 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:38.729567 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:38.730682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:38.730661 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:13:38.972864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:38.972833 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:38.973694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:38.973667 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:38.974418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:38.974379 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:13:53.296628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:53.289975 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79"] Oct 30 19:13:53.296628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:53.290243 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" containerID="cri-o://c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c" gracePeriod=30 Oct 30 19:13:57.783106 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:57.783068 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:13:58.813114 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.813038 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:13:58.951819 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.951599 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-envoy\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.952454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.951767 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-kube-api-access-rqjrw\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-kube-api-access-rqjrw\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.952885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.952666 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-credential-socket\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.952885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.952713 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-certs\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.952885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.952795 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-token\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-token\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.953168 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.953149 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-podinfo\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.953256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.953198 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-data\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.953463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.953379 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-socket\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.953463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.953433 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16fb92f-e2df-4da5-b636-ea5f677da940-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16fb92f-e2df-4da5-b636-ea5f677da940-istiod-ca-cert\") pod \"c16fb92f-e2df-4da5-b636-ea5f677da940\" (UID: \"c16fb92f-e2df-4da5-b636-ea5f677da940\") " Oct 30 19:13:58.953791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.953765 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-certs" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:58.955122 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.955094 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:58.955978 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.955951 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16fb92f-e2df-4da5-b636-ea5f677da940-istiod-ca-cert" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:58.956711 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.956682 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-envoy" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:58.957295 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.957260 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-data" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:58.957295 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.957270 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-socket" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:58.957441 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.957381 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-credential-socket" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:58.957781 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.957760 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-token" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:58.959773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.959741 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-kube-api-access-rqjrw" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "kube-api-access-rqjrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:58.959863 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:58.959843 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-podinfo" pod "c16fb92f-e2df-4da5-b636-ea5f677da940" (UID: "c16fb92f-e2df-4da5-b636-ea5f677da940"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:59.056743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.056709 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:59.056743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.056739 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:59.056743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.056748 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:59.057016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.056760 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16fb92f-e2df-4da5-b636-ea5f677da940-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:59.057016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.056768 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:59.057016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.056777 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqjrw\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-kube-api-access-rqjrw\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:59.057016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.056789 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16fb92f-e2df-4da5-b636-ea5f677da940-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:59.057016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.056802 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16fb92f-e2df-4da5-b636-ea5f677da940-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:59.485244 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.485205 5959 generic.go:356] "Generic (PLEG): container finished" podID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerID="c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c" exitCode=0 Oct 30 19:13:59.485488 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.485294 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" Oct 30 19:13:59.485488 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.485305 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" event={"ID":"c16fb92f-e2df-4da5-b636-ea5f677da940","Type":"ContainerDied","Data":"c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c"} Oct 30 19:13:59.485488 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.485396 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79" event={"ID":"c16fb92f-e2df-4da5-b636-ea5f677da940","Type":"ContainerDied","Data":"ca9dda303431a651ca321ec5729bbeee5ad43c23dc51b3a3328c4df240a7d8ec"} Oct 30 19:13:59.485488 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.485430 5959 scope.go:122] "RemoveContainer" containerID="c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c" Oct 30 19:13:59.500038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.500017 5959 scope.go:122] "RemoveContainer" containerID="c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c" Oct 30 19:13:59.500363 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:13:59.500344 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c\": container with ID starting with c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c not found: ID does not exist" containerID="c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c" Oct 30 19:13:59.500423 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.500371 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c"} err="failed to get container status \"c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c\": rpc error: code = NotFound desc = could not find container \"c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c\": container with ID starting with c7d7023808732ccfcec664ee3ed012bfed6319ee055406acba7acff119f6e70c not found: ID does not exist" Oct 30 19:13:59.511646 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.511599 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79"] Oct 30 19:13:59.517976 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.517950 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-bbj79"] Oct 30 19:13:59.800914 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:13:59.800839 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" path="/var/lib/kubelet/pods/c16fb92f-e2df-4da5-b636-ea5f677da940/volumes" Oct 30 19:14:01.577311 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.577270 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp"] Oct 30 19:14:01.578472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.578449 5959 cpu_manager.go:414] "containerMap: removing container" podUID="87e0e9ca-04c2-461d-9749-7a31da9847cb" containerName="payload-processing" Oct 30 19:14:01.578472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.578473 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e0e9ca-04c2-461d-9749-7a31da9847cb" containerName="payload-processing" Oct 30 19:14:01.578472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.578486 5959 cpu_manager.go:414] "containerMap: removing container" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" Oct 30 19:14:01.578726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.578491 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" Oct 30 19:14:01.578726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.578649 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e0e9ca-04c2-461d-9749-7a31da9847cb" containerName="payload-processing" Oct 30 19:14:01.578726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.578659 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16fb92f-e2df-4da5-b636-ea5f677da940" containerName="istio-proxy" Oct 30 19:14:01.584793 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.584770 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.601348 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.601431 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.601514 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54xw2\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-kube-api-access-54xw2\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.601634 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.601677 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.601805 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.601837 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.601986 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.602407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.602105 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.609664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.608540 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp"] Oct 30 19:14:01.668437 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.668409 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d6f956-openshift-default-dockercfg-jmqmf\"" Oct 30 19:14:01.704052 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704016 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704254 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704075 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704254 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704128 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704254 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704193 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54xw2\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-kube-api-access-54xw2\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704534 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704276 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704534 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704317 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704534 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704427 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704534 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704462 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704534 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704469 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704876 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704644 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704876 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704736 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704876 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704788 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.704981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.704948 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.705678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.705653 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.707101 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.707080 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.707189 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.707088 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.712513 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.712484 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.712619 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.712590 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54xw2\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-kube-api-access-54xw2\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:01.910229 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:01.910194 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:02.230445 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:02.230401 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:02.230576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:02.230495 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:02.230576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:02.230536 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:02.230855 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:02.230825 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp"] Oct 30 19:14:02.513801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:02.513701 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" event={"ID":"c654a045-0394-4c1f-a8fe-a67f9e22ea2e","Type":"ContainerStarted","Data":"03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2"} Oct 30 19:14:02.513801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:02.513767 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" event={"ID":"c654a045-0394-4c1f-a8fe-a67f9e22ea2e","Type":"ContainerStarted","Data":"a56409a247799df403297cddcfa66cf8126d2539bb9f1edeeb0327a0500ae857"} Oct 30 19:14:02.536482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:02.536419 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podStartSLOduration=1.536404089 podStartE2EDuration="1.536404089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:02.532598246 +0000 UTC m=+2222.861009633" watchObservedRunningTime="2026-10-30 19:14:02.536404089 +0000 UTC m=+2222.864815474" Oct 30 19:14:02.910433 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:02.910389 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:03.917034 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:03.916994 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 19:14:03.917691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:03.917645 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 19:14:04.919642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:04.918788 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 19:14:04.919642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:04.918872 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 19:14:05.920861 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:05.919991 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 19:14:05.920861 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:05.920055 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 19:14:06.366366 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:06.366264 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:14:06.922284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:06.922244 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 19:14:06.922799 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:06.922336 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 19:14:07.933629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:07.929024 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 19:14:07.933629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:07.929088 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 19:14:08.917873 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:08.917834 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 19:14:08.918209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:08.918179 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 19:14:09.918288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:09.918258 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:09.918748 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:09.918706 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:09.919876 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:09.919846 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:14:14.125773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:14.119186 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5"] Oct 30 19:14:14.125773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:14.124947 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" containerID="cri-o://6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f" gracePeriod=30 Oct 30 19:14:19.417217 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.417136 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:14:19.447753 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.447714 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-data\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.447947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.447812 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-kube-api-access-842w5\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-kube-api-access-842w5\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.447947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.447896 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-credential-socket\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.448070 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.448019 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-certs\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.448128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.448066 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-envoy\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.448128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.448105 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-token\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-token\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.448249 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.448116 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-data" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:19.448318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.448301 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-podinfo\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.448381 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.448329 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-credential-socket" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:19.448381 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.448344 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-socket\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.448541 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.448405 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istiod-ca-cert\") pod \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\" (UID: \"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d\") " Oct 30 19:14:19.449337 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.449310 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.449481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.449341 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.449481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.449381 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-certs" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:19.449481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.449452 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-socket" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:19.449870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.449846 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istiod-ca-cert" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:19.451403 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.451367 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-token" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:19.452332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.452295 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-envoy" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:19.452441 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.452358 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-podinfo" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:19.453818 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.453796 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr"] Oct 30 19:14:19.454169 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.454149 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-kube-api-access-842w5" pod "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" (UID: "b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d"). InnerVolumeSpecName "kube-api-access-842w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:19.454670 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.454655 5959 cpu_manager.go:414] "containerMap: removing container" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" Oct 30 19:14:19.454726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.454671 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" Oct 30 19:14:19.454843 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.454833 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" Oct 30 19:14:19.454880 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.454850 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" Oct 30 19:14:19.461977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.461950 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.470332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.470297 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr"] Oct 30 19:14:19.550945 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.550910 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.550945 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.550955 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.551163 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551077 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.551163 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551108 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.551245 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551214 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.551285 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551241 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrbr\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-kube-api-access-gzrbr\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.551285 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551269 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.551351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551287 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.551425 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551413 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.551564 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551553 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.551599 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551567 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-842w5\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-kube-api-access-842w5\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.551599 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551578 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.551599 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551587 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.551599 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551596 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.551599 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551620 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.551599 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.551629 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:19.652813 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.652776 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.652813 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.652819 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653032 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.652881 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653032 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.652900 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrbr\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-kube-api-access-gzrbr\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653103 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653045 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653103 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653094 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653264 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653246 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653328 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653296 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653430 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653508 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653476 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653816 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653791 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653916 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653849 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.653916 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.653907 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.654320 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.654295 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.655336 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.655316 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.655746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.655727 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.660996 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.660973 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.661176 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.661156 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrbr\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-kube-api-access-gzrbr\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:19.743726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.741757 5959 generic.go:356] "Generic (PLEG): container finished" podID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerID="6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f" exitCode=0 Oct 30 19:14:19.743726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.741815 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" Oct 30 19:14:19.743726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.741852 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" event={"ID":"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d","Type":"ContainerDied","Data":"6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f"} Oct 30 19:14:19.743726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.741921 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5" event={"ID":"b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d","Type":"ContainerDied","Data":"86b80675215b46b083aa2c1627c3e2e958157bbda6621d55afbfa4f7c0ce9693"} Oct 30 19:14:19.743726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.741947 5959 scope.go:122] "RemoveContainer" containerID="6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f" Oct 30 19:14:19.759557 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.759538 5959 scope.go:122] "RemoveContainer" containerID="d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050" Oct 30 19:14:19.771822 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.771793 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5"] Oct 30 19:14:19.776158 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.776137 5959 scope.go:122] "RemoveContainer" containerID="6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f" Oct 30 19:14:19.776564 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:19.776541 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f\": container with ID starting with 6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f not found: ID does not exist" containerID="6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f" Oct 30 19:14:19.776680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.776578 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f"} err="failed to get container status \"6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f\": rpc error: code = NotFound desc = could not find container \"6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f\": container with ID starting with 6f36292d4c84061aa665fe0c06667bedbdf21c3e546d1d874d7f27472fe7753f not found: ID does not exist" Oct 30 19:14:19.776680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.776618 5959 scope.go:122] "RemoveContainer" containerID="d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050" Oct 30 19:14:19.776942 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:19.776920 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050\": container with ID starting with d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050 not found: ID does not exist" containerID="d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050" Oct 30 19:14:19.776990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.776950 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050"} err="failed to get container status \"d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050\": rpc error: code = NotFound desc = could not find container \"d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050\": container with ID starting with d7fb156150a24ee10c968f733ba3fb9ae852304b6f931070fddd3319da7c4050 not found: ID does not exist" Oct 30 19:14:19.777856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.777837 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-37f2b8-openshift-default-8fdbcc5c7-clns5"] Oct 30 19:14:19.799470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.799440 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" path="/var/lib/kubelet/pods/b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d/volumes" Oct 30 19:14:19.815779 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:19.815745 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:20.143587 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:14:20.143550 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ceedfeb_9369_4ad4_bb4a_d23c669a6e34.slice/crio-020f9eec948b57ac4020ee3a738f39159646e3e6d4ff2505ac7f028ec01f3914 WatchSource:0}: Error finding container 020f9eec948b57ac4020ee3a738f39159646e3e6d4ff2505ac7f028ec01f3914: Status 404 returned error can't find the container with id 020f9eec948b57ac4020ee3a738f39159646e3e6d4ff2505ac7f028ec01f3914 Oct 30 19:14:20.143823 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:20.143805 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr"] Oct 30 19:14:20.146851 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:20.146809 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:20.146970 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:20.146884 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:20.146970 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:20.146915 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:20.768712 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:20.768665 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" event={"ID":"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34","Type":"ContainerStarted","Data":"05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5"} Oct 30 19:14:20.769362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:20.768733 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" event={"ID":"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34","Type":"ContainerStarted","Data":"020f9eec948b57ac4020ee3a738f39159646e3e6d4ff2505ac7f028ec01f3914"} Oct 30 19:14:20.792399 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:20.792347 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podStartSLOduration=4.792332254 podStartE2EDuration="4.792332254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:20.787847047 +0000 UTC m=+2241.116258433" watchObservedRunningTime="2026-10-30 19:14:20.792332254 +0000 UTC m=+2241.120743639" Oct 30 19:14:20.817632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:20.816877 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:21.822634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:21.821887 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 19:14:21.822634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:21.821948 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 19:14:22.823634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:22.821945 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 19:14:22.823634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:22.822004 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 19:14:23.261641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:23.260418 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4"] Oct 30 19:14:23.818635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:23.817855 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 19:14:23.818635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:23.817960 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 19:14:24.346343 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.345956 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2"] Oct 30 19:14:24.347150 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.347124 5959 cpu_manager.go:414] "containerMap: removing container" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" Oct 30 19:14:24.347346 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.347328 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36f4f6a-e4fd-48e8-9e96-04d5a65fc49d" containerName="istio-proxy" Oct 30 19:14:24.352402 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.352382 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.359722 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.359690 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2"] Oct 30 19:14:24.363474 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.363447 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"agc-cred-fcfd33bd0bc0\"" Oct 30 19:14:24.363795 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.363777 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e51c10\"/\"payload-processing-external-model-e2e--24cd22ea-dockercfg-755pg\"" Oct 30 19:14:24.453704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.453656 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2"] Oct 30 19:14:24.454748 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:24.454592 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vtnsr plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" podUID="b73854f5-9346-41c8-9d92-0ae94505a753" Oct 30 19:14:24.491419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.488251 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtnsr\" (UniqueName: \"kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-kube-api-access-vtnsr\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.491419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.488350 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.491419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.488409 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.491419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.488566 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.590377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.590336 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.590597 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.590538 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtnsr\" (UniqueName: \"kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-kube-api-access-vtnsr\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.590597 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.590587 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.590798 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.590669 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.590798 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:24.590773 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e51c10/agc-cred-fcfd33bd0bc0: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:14:24.590798 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:24.590791 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:14:24.590978 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:24.590808 5959 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-plugins-e2e-extmod-a-e51c10: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e51c10" not found Oct 30 19:14:24.590978 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:24.590877 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-provider-credentials podName:b73854f5-9346-41c8-9d92-0ae94505a753 nodeName:}" failed. No retries permitted until 2026-10-30 19:14:25.090858267 +0000 UTC m=+2245.419269634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" (UID: "b73854f5-9346-41c8-9d92-0ae94505a753") : secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:14:24.590978 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:24.590893 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-plugins-config-volume podName:b73854f5-9346-41c8-9d92-0ae94505a753 nodeName:}" failed. No retries permitted until 2026-10-30 19:14:25.090886319 +0000 UTC m=+2245.419297682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" (UID: "b73854f5-9346-41c8-9d92-0ae94505a753") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e51c10" not found Oct 30 19:14:24.591240 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.591216 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.607526 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:24.607023 5959 projected.go:196] Error preparing data for projected volume kube-api-access-vtnsr for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2: failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-e51c10" not found Oct 30 19:14:24.607526 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:24.607115 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-kube-api-access-vtnsr podName:b73854f5-9346-41c8-9d92-0ae94505a753 nodeName:}" failed. No retries permitted until 2026-10-30 19:14:25.107091494 +0000 UTC m=+2245.435502875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vtnsr" (UniqueName: "kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-kube-api-access-vtnsr") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" (UID: "b73854f5-9346-41c8-9d92-0ae94505a753") : failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-e51c10" not found Oct 30 19:14:24.829674 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.819040 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 19:14:24.829674 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.819104 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 19:14:24.919827 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.919788 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:24.949705 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:24.949675 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:25.044951 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.044917 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-routing-overlay-volume\") pod \"b73854f5-9346-41c8-9d92-0ae94505a753\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " Oct 30 19:14:25.045274 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.045249 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-routing-overlay-volume" pod "b73854f5-9346-41c8-9d92-0ae94505a753" (UID: "b73854f5-9346-41c8-9d92-0ae94505a753"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:25.045754 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.045738 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:25.147374 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.147341 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtnsr\" (UniqueName: \"kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-kube-api-access-vtnsr\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:25.147587 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.147406 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:25.147587 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.147449 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2\" (UID: \"b73854f5-9346-41c8-9d92-0ae94505a753\") " pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:25.147587 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:25.147563 5959 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e51c10/agc-cred-fcfd33bd0bc0: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:14:25.147792 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:25.147591 5959 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2: secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:14:25.147792 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:25.147627 5959 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-plugins-e2e-extmod-a-e51c10: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e51c10" not found Oct 30 19:14:25.147792 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:25.147693 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-plugins-config-volume podName:b73854f5-9346-41c8-9d92-0ae94505a753 nodeName:}" failed. No retries permitted until 2026-10-30 19:14:26.147675608 +0000 UTC m=+2246.476086971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" (UID: "b73854f5-9346-41c8-9d92-0ae94505a753") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e51c10" not found Oct 30 19:14:25.147792 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:25.147710 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-provider-credentials podName:b73854f5-9346-41c8-9d92-0ae94505a753 nodeName:}" failed. No retries permitted until 2026-10-30 19:14:26.147703978 +0000 UTC m=+2246.476115340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" (UID: "b73854f5-9346-41c8-9d92-0ae94505a753") : secret "agc-cred-fcfd33bd0bc0" not found Oct 30 19:14:25.152890 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:25.152702 5959 projected.go:196] Error preparing data for projected volume kube-api-access-vtnsr for pod ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2: failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-e51c10" not found Oct 30 19:14:25.152890 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:25.152794 5959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-kube-api-access-vtnsr podName:b73854f5-9346-41c8-9d92-0ae94505a753 nodeName:}" failed. No retries permitted until 2026-10-30 19:14:26.152772606 +0000 UTC m=+2246.481183983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtnsr" (UniqueName: "kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-kube-api-access-vtnsr") pod "payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" (UID: "b73854f5-9346-41c8-9d92-0ae94505a753") : failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-e51c10" not found Oct 30 19:14:25.158835 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.158807 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn"] Oct 30 19:14:25.817270 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.816739 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 19:14:25.817270 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.816800 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 19:14:25.932999 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.932963 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2" Oct 30 19:14:25.967696 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.967592 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2"] Oct 30 19:14:25.972181 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:25.972147 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5dc8m58k2"] Oct 30 19:14:26.088875 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:26.086353 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:26.088875 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:26.086398 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtnsr\" (UniqueName: \"kubernetes.io/projected/b73854f5-9346-41c8-9d92-0ae94505a753-kube-api-access-vtnsr\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:26.088875 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:26.086418 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73854f5-9346-41c8-9d92-0ae94505a753-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:26.820633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:26.820089 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 19:14:26.820633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:26.820156 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 19:14:27.789772 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:27.789727 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:14:27.801965 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:27.801931 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73854f5-9346-41c8-9d92-0ae94505a753" path="/var/lib/kubelet/pods/b73854f5-9346-41c8-9d92-0ae94505a753/volumes" Oct 30 19:14:27.822524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:27.822495 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:27.823012 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:27.822845 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:27.823706 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:27.823686 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:14:28.261774 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.261732 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" podUID="bd5c26a7-8604-42f9-9372-b887480f173d" containerName="payload-processing" containerID="cri-o://48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1" gracePeriod=25 Oct 30 19:14:28.570679 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.570636 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:14:28.604068 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.604036 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5"] Oct 30 19:14:28.605128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.605103 5959 cpu_manager.go:414] "containerMap: removing container" podUID="bd5c26a7-8604-42f9-9372-b887480f173d" containerName="payload-processing" Oct 30 19:14:28.605128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.605127 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5c26a7-8604-42f9-9372-b887480f173d" containerName="payload-processing" Oct 30 19:14:28.605357 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.605342 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5c26a7-8604-42f9-9372-b887480f173d" containerName="payload-processing" Oct 30 19:14:28.609509 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.609490 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.618526 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.618491 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5"] Oct 30 19:14:28.760834 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.759758 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-provider-credentials\") pod \"bd5c26a7-8604-42f9-9372-b887480f173d\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " Oct 30 19:14:28.760834 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.760425 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-plugins-config-volume\") pod \"bd5c26a7-8604-42f9-9372-b887480f173d\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " Oct 30 19:14:28.761107 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.760897 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-plugins-config-volume" pod "bd5c26a7-8604-42f9-9372-b887480f173d" (UID: "bd5c26a7-8604-42f9-9372-b887480f173d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:28.761107 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.761039 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-kube-api-access-mb66c\") pod \"bd5c26a7-8604-42f9-9372-b887480f173d\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " Oct 30 19:14:28.761268 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.761219 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-routing-overlay-volume\") pod \"bd5c26a7-8604-42f9-9372-b887480f173d\" (UID: \"bd5c26a7-8604-42f9-9372-b887480f173d\") " Oct 30 19:14:28.762565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.762528 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-routing-overlay-volume" pod "bd5c26a7-8604-42f9-9372-b887480f173d" (UID: "bd5c26a7-8604-42f9-9372-b887480f173d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:28.762773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.762711 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.762849 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.762799 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.762902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.762874 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.762954 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.762907 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.763262 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.763219 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxt2x\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-kube-api-access-bxt2x\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.763410 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.763378 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.763516 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.763481 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.763579 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.763559 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.763694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.763593 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.763948 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.763926 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:28.764007 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.763956 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd5c26a7-8604-42f9-9372-b887480f173d-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:28.764745 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.764695 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-provider-credentials" pod "bd5c26a7-8604-42f9-9372-b887480f173d" (UID: "bd5c26a7-8604-42f9-9372-b887480f173d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:28.765715 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.765692 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-kube-api-access-mb66c" pod "bd5c26a7-8604-42f9-9372-b887480f173d" (UID: "bd5c26a7-8604-42f9-9372-b887480f173d"). InnerVolumeSpecName "kube-api-access-mb66c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:28.865664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.865622 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.865664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.865668 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.865791 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.865831 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.865886 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.865915 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866053 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxt2x\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-kube-api-access-bxt2x\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866161 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866178 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866229 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866246 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866275 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866362 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:28.866687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866380 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/bd5c26a7-8604-42f9-9372-b887480f173d-kube-api-access-mb66c\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:28.866687 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866584 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.866977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.866954 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.868367 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.868345 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.868694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.868675 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.874418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.874389 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.874537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.874520 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxt2x\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-kube-api-access-bxt2x\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.936190 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.936150 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:28.965382 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.965346 5959 generic.go:356] "Generic (PLEG): container finished" podID="bd5c26a7-8604-42f9-9372-b887480f173d" containerID="48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1" exitCode=0 Oct 30 19:14:28.965576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.965396 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" Oct 30 19:14:28.965576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.965428 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" event={"ID":"bd5c26a7-8604-42f9-9372-b887480f173d","Type":"ContainerDied","Data":"48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1"} Oct 30 19:14:28.966053 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.966027 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4" event={"ID":"bd5c26a7-8604-42f9-9372-b887480f173d","Type":"ContainerDied","Data":"c9af29e26b57fd4151821bc7b478bd39866dc7c3dbc79b8bd6703f9e08fbf314"} Oct 30 19:14:28.966148 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.966093 5959 scope.go:122] "RemoveContainer" containerID="48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1" Oct 30 19:14:28.980056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.980033 5959 scope.go:122] "RemoveContainer" containerID="48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1" Oct 30 19:14:28.980384 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:28.980360 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1\": container with ID starting with 48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1 not found: ID does not exist" containerID="48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1" Oct 30 19:14:28.980486 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:28.980392 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1"} err="failed to get container status \"48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1\": rpc error: code = NotFound desc = could not find container \"48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1\": container with ID starting with 48db85098d9a8c59edda37d50c9249628f9dc29a2be0837711ab8197d4ad36a1 not found: ID does not exist" Oct 30 19:14:29.008018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.007560 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4"] Oct 30 19:14:29.012072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.012036 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e51c10/payload-processing-external-model-e2e-extmod-a-e51c10-5846gd7x4"] Oct 30 19:14:29.251265 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:14:29.251231 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7b9b00_3bc3_4ece_8926_41355d0a9b5e.slice/crio-faf027692ebfeaf74a1a52cd325a23350df4d823cc59003775bb9841a32a00d3 WatchSource:0}: Error finding container faf027692ebfeaf74a1a52cd325a23350df4d823cc59003775bb9841a32a00d3: Status 404 returned error can't find the container with id faf027692ebfeaf74a1a52cd325a23350df4d823cc59003775bb9841a32a00d3 Oct 30 19:14:29.251738 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.251710 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5"] Oct 30 19:14:29.254268 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.254232 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:29.254351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.254301 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:29.254351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.254333 5959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 19:14:29.800842 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.800809 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5c26a7-8604-42f9-9372-b887480f173d" path="/var/lib/kubelet/pods/bd5c26a7-8604-42f9-9372-b887480f173d/volumes" Oct 30 19:14:29.976793 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.976749 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" event={"ID":"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e","Type":"ContainerStarted","Data":"8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97"} Oct 30 19:14:29.977252 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.976808 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" event={"ID":"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e","Type":"ContainerStarted","Data":"faf027692ebfeaf74a1a52cd325a23350df4d823cc59003775bb9841a32a00d3"} Oct 30 19:14:29.999047 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:29.998979 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" podStartSLOduration=13.998964616 podStartE2EDuration="13.998964616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:29.997037984 +0000 UTC m=+2250.325449382" watchObservedRunningTime="2026-10-30 19:14:29.998964616 +0000 UTC m=+2250.327376022" Oct 30 19:14:30.159184 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.159127 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" podUID="1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73" containerName="payload-processing" containerID="cri-o://bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0" gracePeriod=25 Oct 30 19:14:30.458018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.457987 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:14:30.504709 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.504675 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-kube-api-access-mvqxh\" (UniqueName: \"kubernetes.io/projected/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-kube-api-access-mvqxh\") pod \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\" (UID: \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\") " Oct 30 19:14:30.504924 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.504819 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-plugins-config-volume\") pod \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\" (UID: \"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73\") " Oct 30 19:14:30.505318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.505284 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-plugins-config-volume" pod "1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73" (UID: "1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:30.505831 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.505810 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:30.507833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.507794 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-kube-api-access-mvqxh" pod "1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73" (UID: "1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73"). InnerVolumeSpecName "kube-api-access-mvqxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:30.608002 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.607966 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvqxh\" (UniqueName: \"kubernetes.io/projected/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73-kube-api-access-mvqxh\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:30.952628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.951922 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:30.976629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.974725 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:14:30.976629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:30.974847 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:14:31.055626 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.050992 5959 generic.go:356] "Generic (PLEG): container finished" podID="1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73" containerID="bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0" exitCode=0 Oct 30 19:14:31.055626 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.052280 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" Oct 30 19:14:31.068071 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.068030 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" event={"ID":"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73","Type":"ContainerDied","Data":"bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0"} Oct 30 19:14:31.068440 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.068416 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn" event={"ID":"1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73","Type":"ContainerDied","Data":"2862e091930e909669b0b64d0f6b5de002b0919dde7d4ca3fc1f4e3896206e50"} Oct 30 19:14:31.068631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.068599 5959 scope.go:122] "RemoveContainer" containerID="bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0" Oct 30 19:14:31.145999 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.145971 5959 scope.go:122] "RemoveContainer" containerID="bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0" Oct 30 19:14:31.146309 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:31.146289 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0\": container with ID starting with bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0 not found: ID does not exist" containerID="bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0" Oct 30 19:14:31.146371 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.146317 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0"} err="failed to get container status \"bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0\": rpc error: code = NotFound desc = could not find container \"bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0\": container with ID starting with bce66948f0d2bd1e709746c9f320550164196ce393fc001b2f283ff54fc43ee0 not found: ID does not exist" Oct 30 19:14:31.156657 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.156618 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn"] Oct 30 19:14:31.162135 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.162111 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ecc266-76b554bcc6-ttpkn"] Oct 30 19:14:31.828953 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.828918 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73" path="/var/lib/kubelet/pods/1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73/volumes" Oct 30 19:14:31.937507 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.937474 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:14:31.937744 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:31.937532 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:14:32.937194 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:32.937159 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:14:32.937649 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:32.937217 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:14:33.945476 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:33.945434 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:14:33.946478 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:33.946079 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:14:34.937145 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:34.937114 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:14:34.937339 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:34.937171 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:14:35.936959 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:35.936926 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:14:35.937397 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:35.936990 5959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:14:37.363522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:37.363476 5959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:37.364456 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:37.364412 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:37.365428 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:37.365403 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:14:44.895332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:44.895114 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b"] Oct 30 19:14:44.895833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:44.895441 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" containerID="cri-o://d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956" gracePeriod=30 Oct 30 19:14:44.909594 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:44.909556 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d"] Oct 30 19:14:44.909963 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:44.909925 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74"] Oct 30 19:14:44.910424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:44.910396 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" containerID="cri-o://33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6" gracePeriod=30 Oct 30 19:14:44.910913 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:44.910863 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" containerID="cri-o://1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72" gracePeriod=30 Oct 30 19:14:50.232585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.232546 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:14:50.295902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.295870 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-certs\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.296220 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.296200 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-credential-socket\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.296349 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.296333 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-socket\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.296454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.296441 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-podinfo\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.296770 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.296748 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-kube-api-access-8l2fz\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-kube-api-access-8l2fz\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.296930 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.296911 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-token\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-token\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.297169 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.297151 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istiod-ca-cert\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.297447 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.297432 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-envoy\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.297552 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.297539 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-data\") pod \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\" (UID: \"2e262f71-a759-46c9-8bc6-99f47b2fbc63\") " Oct 30 19:14:50.298893 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.298571 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-data" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.298893 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.298803 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-credential-socket" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.299101 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.299020 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-socket" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.300416 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.300386 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istiod-ca-cert" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:50.304312 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.304286 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-kube-api-access-8l2fz" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "kube-api-access-8l2fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:50.306826 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.306800 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-certs" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.318683 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.318652 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-podinfo" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:50.318838 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.318819 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-token" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:50.321412 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.321373 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-envoy" pod "2e262f71-a759-46c9-8bc6-99f47b2fbc63" (UID: "2e262f71-a759-46c9-8bc6-99f47b2fbc63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.324664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.324461 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:14:50.368761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.367510 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:14:50.399663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.399628 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-data\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.399866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.399749 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-token\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-token\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.399866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.399820 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97adfae8-ff7c-454b-b04b-72d64bb564b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97adfae8-ff7c-454b-b04b-72d64bb564b6-istiod-ca-cert\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.399995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.399922 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-podinfo\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.400052 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.399997 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-kube-api-access-tzdxg\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-kube-api-access-tzdxg\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.400107 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.400068 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-credential-socket\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.400155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.400145 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-certs\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.400262 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.400241 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-envoy\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.401776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.400974 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-certs" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.401776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.401156 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-credential-socket" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.401776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.401524 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97adfae8-ff7c-454b-b04b-72d64bb564b6-istiod-ca-cert" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:50.401776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.401742 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-data" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.402169 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.402152 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-socket\") pod \"97adfae8-ff7c-454b-b04b-72d64bb564b6\" (UID: \"97adfae8-ff7c-454b-b04b-72d64bb564b6\") " Oct 30 19:14:50.403821 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.402495 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-socket" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.404293 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.404265 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-token" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:50.405048 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405028 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405050 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405060 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405068 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405076 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405088 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405104 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e262f71-a759-46c9-8bc6-99f47b2fbc63-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405116 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405125 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97adfae8-ff7c-454b-b04b-72d64bb564b6-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405133 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l2fz\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-kube-api-access-8l2fz\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405535 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405142 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405535 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405150 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405535 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405159 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405535 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405168 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e262f71-a759-46c9-8bc6-99f47b2fbc63-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.405535 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.405176 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.409873 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.409799 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-podinfo" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:50.409873 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.409836 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-kube-api-access-tzdxg" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "kube-api-access-tzdxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:50.411316 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.411295 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-envoy" pod "97adfae8-ff7c-454b-b04b-72d64bb564b6" (UID: "97adfae8-ff7c-454b-b04b-72d64bb564b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.443179 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.443078 5959 generic.go:356] "Generic (PLEG): container finished" podID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerID="33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6" exitCode=0 Oct 30 19:14:50.443179 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.443144 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" Oct 30 19:14:50.443548 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.443160 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" event={"ID":"97adfae8-ff7c-454b-b04b-72d64bb564b6","Type":"ContainerDied","Data":"33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6"} Oct 30 19:14:50.443903 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.443881 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74" event={"ID":"97adfae8-ff7c-454b-b04b-72d64bb564b6","Type":"ContainerDied","Data":"8deeb0a25e180027c994fe009cfdb23d99408cc2e3d316ca9218d001ac568b09"} Oct 30 19:14:50.444019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.443931 5959 scope.go:122] "RemoveContainer" containerID="33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6" Oct 30 19:14:50.445879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.445298 5959 generic.go:356] "Generic (PLEG): container finished" podID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerID="d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956" exitCode=0 Oct 30 19:14:50.445879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.445365 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" Oct 30 19:14:50.445879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.445434 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" event={"ID":"2e262f71-a759-46c9-8bc6-99f47b2fbc63","Type":"ContainerDied","Data":"d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956"} Oct 30 19:14:50.445879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.445475 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b" event={"ID":"2e262f71-a759-46c9-8bc6-99f47b2fbc63","Type":"ContainerDied","Data":"5e0f4e10f144d26382dc00db9f6ea6a0d0db7e621c529cfa4c82311180b3f73a"} Oct 30 19:14:50.447491 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.447466 5959 generic.go:356] "Generic (PLEG): container finished" podID="d9ae6370-0587-43e9-88d0-96594764269f" containerID="1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72" exitCode=0 Oct 30 19:14:50.447747 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.447726 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" event={"ID":"d9ae6370-0587-43e9-88d0-96594764269f","Type":"ContainerDied","Data":"1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72"} Oct 30 19:14:50.447864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.447786 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" event={"ID":"d9ae6370-0587-43e9-88d0-96594764269f","Type":"ContainerDied","Data":"28b1a89c9480d1341233652674d621d7b5bc04f826387df30e9dd8b2334de4c9"} Oct 30 19:14:50.447864 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.447858 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d" Oct 30 19:14:50.459414 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.459392 5959 scope.go:122] "RemoveContainer" containerID="33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6" Oct 30 19:14:50.459770 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:50.459749 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6\": container with ID starting with 33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6 not found: ID does not exist" containerID="33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6" Oct 30 19:14:50.459872 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.459790 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6"} err="failed to get container status \"33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6\": rpc error: code = NotFound desc = could not find container \"33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6\": container with ID starting with 33e313325722da1c22a645f9c69e97b98767c80f6059d158da29660820242ef6 not found: ID does not exist" Oct 30 19:14:50.459872 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.459813 5959 scope.go:122] "RemoveContainer" containerID="d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956" Oct 30 19:14:50.472523 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.472503 5959 scope.go:122] "RemoveContainer" containerID="d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956" Oct 30 19:14:50.472861 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:50.472839 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956\": container with ID starting with d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956 not found: ID does not exist" containerID="d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956" Oct 30 19:14:50.472957 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.472873 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956"} err="failed to get container status \"d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956\": rpc error: code = NotFound desc = could not find container \"d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956\": container with ID starting with d2474cdb05466351c6ca0b6ea77cfbe0d6c4dbb78775b39e32cb64d057c6c956 not found: ID does not exist" Oct 30 19:14:50.472957 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.472898 5959 scope.go:122] "RemoveContainer" containerID="1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72" Oct 30 19:14:50.473359 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.473340 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74"] Oct 30 19:14:50.479661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.479635 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-mzg74"] Oct 30 19:14:50.485850 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.485829 5959 scope.go:122] "RemoveContainer" containerID="1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72" Oct 30 19:14:50.486768 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:50.486735 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72\": container with ID starting with 1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72 not found: ID does not exist" containerID="1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72" Oct 30 19:14:50.486904 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.486785 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72"} err="failed to get container status \"1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72\": rpc error: code = NotFound desc = could not find container \"1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72\": container with ID starting with 1b39bfd7b93d880dad05df037a6b2b3b724a47cebbf0fc14d22c74392efcbe72 not found: ID does not exist" Oct 30 19:14:50.491742 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.491707 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b"] Oct 30 19:14:50.495865 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.495845 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-rg24b"] Oct 30 19:14:50.505998 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.505972 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9ae6370-0587-43e9-88d0-96594764269f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ae6370-0587-43e9-88d0-96594764269f-istio-podinfo\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.506116 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506032 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-certs\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.506116 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506074 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-istio-token\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-istio-token\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.506224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506114 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-envoy\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.506281 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506262 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-certs" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.506422 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506399 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-credential-socket\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.506545 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506527 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ae6370-0587-43e9-88d0-96594764269f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ae6370-0587-43e9-88d0-96594764269f-istiod-ca-cert\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.506618 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506571 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-data\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.506726 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506707 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-credential-socket" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.506797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506713 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-socket\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.506903 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506884 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-kube-api-access-gkdxz\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-kube-api-access-gkdxz\") pod \"d9ae6370-0587-43e9-88d0-96594764269f\" (UID: \"d9ae6370-0587-43e9-88d0-96594764269f\") " Oct 30 19:14:50.507038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506955 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ae6370-0587-43e9-88d0-96594764269f-istiod-ca-cert" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:50.507038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507002 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-socket" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.507038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.506986 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-data" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.507645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507534 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.507645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507555 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.507645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507569 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzdxg\" (UniqueName: \"kubernetes.io/projected/97adfae8-ff7c-454b-b04b-72d64bb564b6-kube-api-access-tzdxg\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.507645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507584 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ae6370-0587-43e9-88d0-96594764269f-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.507645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507592 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.507645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507643 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.507645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507653 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97adfae8-ff7c-454b-b04b-72d64bb564b6-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.507645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.507661 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.508683 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.508663 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-istio-token" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:50.508834 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.508817 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-envoy" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:50.508834 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.508824 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9ae6370-0587-43e9-88d0-96594764269f-istio-podinfo" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:50.509198 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.509183 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-kube-api-access-gkdxz" pod "d9ae6370-0587-43e9-88d0-96594764269f" (UID: "d9ae6370-0587-43e9-88d0-96594764269f"). InnerVolumeSpecName "kube-api-access-gkdxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:50.609708 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.609671 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkdxz\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-kube-api-access-gkdxz\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.609708 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.609707 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ae6370-0587-43e9-88d0-96594764269f-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.609928 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.609722 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ae6370-0587-43e9-88d0-96594764269f-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.609928 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.609734 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ae6370-0587-43e9-88d0-96594764269f-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.780834 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.780797 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d"] Oct 30 19:14:50.785255 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:50.785218 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-spj2d"] Oct 30 19:14:51.799941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:51.799896 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" path="/var/lib/kubelet/pods/2e262f71-a759-46c9-8bc6-99f47b2fbc63/volumes" Oct 30 19:14:51.800692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:51.800672 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" path="/var/lib/kubelet/pods/97adfae8-ff7c-454b-b04b-72d64bb564b6/volumes" Oct 30 19:14:51.801233 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:51.801212 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ae6370-0587-43e9-88d0-96594764269f" path="/var/lib/kubelet/pods/d9ae6370-0587-43e9-88d0-96594764269f/volumes" Oct 30 19:14:53.591504 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:53.590001 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5"] Oct 30 19:14:55.586091 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:55.586056 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7"] Oct 30 19:14:57.781396 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:57.781352 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:14:58.590713 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:58.590661 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" podUID="45e95592-3e89-49e5-a892-2cd8b5ddcb1f" containerName="payload-processing" containerID="cri-o://762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8" gracePeriod=25 Oct 30 19:14:58.896772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:58.896743 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:14:59.041311 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.041272 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-kube-api-access-29d58\" (UniqueName: \"kubernetes.io/projected/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-kube-api-access-29d58\") pod \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\" (UID: \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\") " Oct 30 19:14:59.041496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.041397 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-plugins-config-volume\") pod \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\" (UID: \"45e95592-3e89-49e5-a892-2cd8b5ddcb1f\") " Oct 30 19:14:59.041816 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.041791 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-plugins-config-volume" pod "45e95592-3e89-49e5-a892-2cd8b5ddcb1f" (UID: "45e95592-3e89-49e5-a892-2cd8b5ddcb1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:59.042106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.042090 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:59.043774 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.043739 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-kube-api-access-29d58" pod "45e95592-3e89-49e5-a892-2cd8b5ddcb1f" (UID: "45e95592-3e89-49e5-a892-2cd8b5ddcb1f"). InnerVolumeSpecName "kube-api-access-29d58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:59.144062 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.144028 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29d58\" (UniqueName: \"kubernetes.io/projected/45e95592-3e89-49e5-a892-2cd8b5ddcb1f-kube-api-access-29d58\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:59.583089 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.582982 5959 generic.go:356] "Generic (PLEG): container finished" podID="45e95592-3e89-49e5-a892-2cd8b5ddcb1f" containerID="762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8" exitCode=0 Oct 30 19:14:59.583319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.583288 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" event={"ID":"45e95592-3e89-49e5-a892-2cd8b5ddcb1f","Type":"ContainerDied","Data":"762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8"} Oct 30 19:14:59.583404 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.583367 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" event={"ID":"45e95592-3e89-49e5-a892-2cd8b5ddcb1f","Type":"ContainerDied","Data":"f4071023c48cb0696885e7e71ed7bd0ab867095c381fa6b49a788517d7a9a4ad"} Oct 30 19:14:59.583468 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.583402 5959 scope.go:122] "RemoveContainer" containerID="762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8" Oct 30 19:14:59.583675 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.583656 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5" Oct 30 19:14:59.604415 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.603396 5959 scope.go:122] "RemoveContainer" containerID="762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8" Oct 30 19:14:59.604415 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:14:59.604311 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8\": container with ID starting with 762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8 not found: ID does not exist" containerID="762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8" Oct 30 19:14:59.604415 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.604350 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8"} err="failed to get container status \"762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8\": rpc error: code = NotFound desc = could not find container \"762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8\": container with ID starting with 762cc72e9ef4bf6213af58601ab0f1042d83343bb6650d2a61ea55640a6bb7d8 not found: ID does not exist" Oct 30 19:14:59.649938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.649871 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5"] Oct 30 19:14:59.657668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.657632 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-baa504-7ff6d9c654-4bns5"] Oct 30 19:14:59.801251 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:14:59.801211 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e95592-3e89-49e5-a892-2cd8b5ddcb1f" path="/var/lib/kubelet/pods/45e95592-3e89-49e5-a892-2cd8b5ddcb1f/volumes" Oct 30 19:15:00.218379 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.218341 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr"] Oct 30 19:15:00.219320 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219299 5959 cpu_manager.go:414] "containerMap: removing container" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" Oct 30 19:15:00.219320 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219318 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" Oct 30 19:15:00.219520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219330 5959 cpu_manager.go:414] "containerMap: removing container" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" Oct 30 19:15:00.219520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219335 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" Oct 30 19:15:00.219520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219376 5959 cpu_manager.go:414] "containerMap: removing container" podUID="1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73" containerName="payload-processing" Oct 30 19:15:00.219520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219381 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73" containerName="payload-processing" Oct 30 19:15:00.219520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219399 5959 cpu_manager.go:414] "containerMap: removing container" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" Oct 30 19:15:00.219520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219404 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" Oct 30 19:15:00.219520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219409 5959 cpu_manager.go:414] "containerMap: removing container" podUID="45e95592-3e89-49e5-a892-2cd8b5ddcb1f" containerName="payload-processing" Oct 30 19:15:00.219520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219414 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e95592-3e89-49e5-a892-2cd8b5ddcb1f" containerName="payload-processing" Oct 30 19:15:00.220033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219535 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebdb22e-e42f-4ec2-be24-ab1e9b0cff73" containerName="payload-processing" Oct 30 19:15:00.220033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219546 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ae6370-0587-43e9-88d0-96594764269f" containerName="istio-proxy" Oct 30 19:15:00.220033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219552 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e95592-3e89-49e5-a892-2cd8b5ddcb1f" containerName="payload-processing" Oct 30 19:15:00.220033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219563 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e262f71-a759-46c9-8bc6-99f47b2fbc63" containerName="istio-proxy" Oct 30 19:15:00.220033 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.219570 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="97adfae8-ff7c-454b-b04b-72d64bb564b6" containerName="istio-proxy" Oct 30 19:15:00.230824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.230791 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4"] Oct 30 19:15:00.230994 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.230918 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:00.236000 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.235974 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:00.241891 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.241857 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr"] Oct 30 19:15:00.257841 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.257380 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4"] Oct 30 19:15:00.260025 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.259998 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\"" Oct 30 19:15:00.268775 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.268743 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk"] Oct 30 19:15:00.281854 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.281821 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:00.296253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.292505 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q"] Oct 30 19:15:00.312067 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.303594 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk"] Oct 30 19:15:00.312067 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.303734 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:00.324626 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.324582 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q"] Oct 30 19:15:00.394350 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.394312 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwqz\" (UniqueName: \"kubernetes.io/projected/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-kube-api-access-8mwqz\") pod \"maas-api-key-cleanup-29889795-rhkfr\" (UID: \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:00.394350 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.394356 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef922752-ebe2-4428-81b5-95e146bff23f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk\" (UID: \"ef922752-ebe2-4428-81b5-95e146bff23f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:00.394594 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.394510 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5t6\" (UniqueName: \"kubernetes.io/projected/ef922752-ebe2-4428-81b5-95e146bff23f-kube-api-access-wl5t6\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk\" (UID: \"ef922752-ebe2-4428-81b5-95e146bff23f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:00.394772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.394754 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4\" (UID: \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:00.394817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.394803 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2h9\" (UniqueName: \"kubernetes.io/projected/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-kube-api-access-7t2h9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4\" (UID: \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:00.394883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.394872 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-rhkfr\" (UID: \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:00.496855 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.496764 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4\" (UID: \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:00.496855 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.496826 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2h9\" (UniqueName: \"kubernetes.io/projected/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-kube-api-access-7t2h9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4\" (UID: \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:00.497095 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.496902 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-rhkfr\" (UID: \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:00.497095 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.496957 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwqz\" (UniqueName: \"kubernetes.io/projected/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-kube-api-access-8mwqz\") pod \"maas-api-key-cleanup-29889795-rhkfr\" (UID: \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:00.497095 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.496990 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0c1081-4209-4c4b-912c-e1defc785003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q\" (UID: \"8c0c1081-4209-4c4b-912c-e1defc785003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:00.497095 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.497071 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef922752-ebe2-4428-81b5-95e146bff23f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk\" (UID: \"ef922752-ebe2-4428-81b5-95e146bff23f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:00.497298 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.497175 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5t6\" (UniqueName: \"kubernetes.io/projected/ef922752-ebe2-4428-81b5-95e146bff23f-kube-api-access-wl5t6\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk\" (UID: \"ef922752-ebe2-4428-81b5-95e146bff23f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:00.497359 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.497347 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752h5\" (UniqueName: \"kubernetes.io/projected/8c0c1081-4209-4c4b-912c-e1defc785003-kube-api-access-752h5\") pod \"maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q\" (UID: \"8c0c1081-4209-4c4b-912c-e1defc785003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:00.497596 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.497559 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4\" (UID: \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:00.497761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.497715 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-rhkfr\" (UID: \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:00.497865 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.497844 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef922752-ebe2-4428-81b5-95e146bff23f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk\" (UID: \"ef922752-ebe2-4428-81b5-95e146bff23f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:00.508808 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.508778 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2h9\" (UniqueName: \"kubernetes.io/projected/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-kube-api-access-7t2h9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4\" (UID: \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:00.512646 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.512592 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwqz\" (UniqueName: \"kubernetes.io/projected/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-kube-api-access-8mwqz\") pod \"maas-api-key-cleanup-29889795-rhkfr\" (UID: \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:00.512822 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.512805 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5t6\" (UniqueName: \"kubernetes.io/projected/ef922752-ebe2-4428-81b5-95e146bff23f-kube-api-access-wl5t6\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk\" (UID: \"ef922752-ebe2-4428-81b5-95e146bff23f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:00.566272 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.566241 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:00.578697 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.578668 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:00.586823 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.586791 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" podUID="1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" containerName="payload-pre-processing" containerID="cri-o://d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6" gracePeriod=25 Oct 30 19:15:00.599100 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.599067 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752h5\" (UniqueName: \"kubernetes.io/projected/8c0c1081-4209-4c4b-912c-e1defc785003-kube-api-access-752h5\") pod \"maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q\" (UID: \"8c0c1081-4209-4c4b-912c-e1defc785003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:00.599325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.599306 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0c1081-4209-4c4b-912c-e1defc785003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q\" (UID: \"8c0c1081-4209-4c4b-912c-e1defc785003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:00.600131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.600106 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0c1081-4209-4c4b-912c-e1defc785003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q\" (UID: \"8c0c1081-4209-4c4b-912c-e1defc785003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:00.610289 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.610250 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-752h5\" (UniqueName: \"kubernetes.io/projected/8c0c1081-4209-4c4b-912c-e1defc785003-kube-api-access-752h5\") pod \"maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q\" (UID: \"8c0c1081-4209-4c4b-912c-e1defc785003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:00.625936 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.625725 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:00.637074 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.637020 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:00.877352 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.877303 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr"] Oct 30 19:15:00.911138 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.911083 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4"] Oct 30 19:15:00.963729 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.963690 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk"] Oct 30 19:15:00.966459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:00.966433 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q"] Oct 30 19:15:00.967875 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:15:00.967846 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef922752_ebe2_4428_81b5_95e146bff23f.slice/crio-cf775a138adafd4fd9bf5f4e039e7ea119e3fec4050f79422019fbca5531ecc7 WatchSource:0}: Error finding container cf775a138adafd4fd9bf5f4e039e7ea119e3fec4050f79422019fbca5531ecc7: Status 404 returned error can't find the container with id cf775a138adafd4fd9bf5f4e039e7ea119e3fec4050f79422019fbca5531ecc7 Oct 30 19:15:01.604627 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.604507 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" event={"ID":"ef922752-ebe2-4428-81b5-95e146bff23f","Type":"ContainerStarted","Data":"077695192f5b56dc36bcdd7cf5356b567a6a42dc20f53ee9b222b81fcb33bac1"} Oct 30 19:15:01.604627 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.604567 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" event={"ID":"ef922752-ebe2-4428-81b5-95e146bff23f","Type":"ContainerStarted","Data":"cf775a138adafd4fd9bf5f4e039e7ea119e3fec4050f79422019fbca5531ecc7"} Oct 30 19:15:01.606301 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.606276 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" event={"ID":"1bb68f2a-2ebc-4def-91d1-c8258f64dc84","Type":"ContainerStarted","Data":"c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9"} Oct 30 19:15:01.606435 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.606331 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" event={"ID":"1bb68f2a-2ebc-4def-91d1-c8258f64dc84","Type":"ContainerStarted","Data":"a7ca780d71c2a8185a58f67836ba610173b7e178d5fd0b0f7f47ae086bb4eacd"} Oct 30 19:15:01.607933 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.607907 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" event={"ID":"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4","Type":"ContainerStarted","Data":"46a4fdc9c3ebffa18dd04d2e5cf5e620a5a9ed927f90fa3a062548d0128dd276"} Oct 30 19:15:01.608068 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.607956 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" event={"ID":"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4","Type":"ContainerStarted","Data":"30cadd5887abb65d8e85197128a60ecba084be35c613a728e18bba161d0eaac5"} Oct 30 19:15:01.609471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.609442 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" event={"ID":"8c0c1081-4209-4c4b-912c-e1defc785003","Type":"ContainerStarted","Data":"7872c6d490c9ae56ff561a185002e475ce98a195585074f5b61668cfd4a71273"} Oct 30 19:15:01.609581 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.609484 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" event={"ID":"8c0c1081-4209-4c4b-912c-e1defc785003","Type":"ContainerStarted","Data":"1c7ccaa838e730d6a15a2cd4f63ec9ddd9bd4f89718cbcfbd36ebbe137b4256e"} Oct 30 19:15:01.624312 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.624258 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" podStartSLOduration=1.6242414090000001 podStartE2EDuration="1.624241409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:01.6225352 +0000 UTC m=+2281.950946585" watchObservedRunningTime="2026-10-30 19:15:01.624241409 +0000 UTC m=+2281.952652824" Oct 30 19:15:01.640069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.640015 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" podStartSLOduration=1.639999584 podStartE2EDuration="1.639999584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:01.637297696 +0000 UTC m=+2281.965709080" watchObservedRunningTime="2026-10-30 19:15:01.639999584 +0000 UTC m=+2281.968410968" Oct 30 19:15:01.654779 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.654727 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" podStartSLOduration=1.654709074 podStartE2EDuration="1.654709074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:01.651776975 +0000 UTC m=+2281.980188373" watchObservedRunningTime="2026-10-30 19:15:01.654709074 +0000 UTC m=+2281.983120460" Oct 30 19:15:01.669165 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:01.669108 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" podStartSLOduration=1.6690919480000002 podStartE2EDuration="1.669091948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:01.667592715 +0000 UTC m=+2281.996004104" watchObservedRunningTime="2026-10-30 19:15:01.669091948 +0000 UTC m=+2281.997503390" Oct 30 19:15:02.620773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.620739 5959 generic.go:356] "Generic (PLEG): container finished" podID="ef922752-ebe2-4428-81b5-95e146bff23f" containerID="077695192f5b56dc36bcdd7cf5356b567a6a42dc20f53ee9b222b81fcb33bac1" exitCode=60 Oct 30 19:15:02.621257 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.620816 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" event={"ID":"ef922752-ebe2-4428-81b5-95e146bff23f","Type":"ContainerDied","Data":"077695192f5b56dc36bcdd7cf5356b567a6a42dc20f53ee9b222b81fcb33bac1"} Oct 30 19:15:02.621257 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.621193 5959 scope.go:122] "RemoveContainer" containerID="077695192f5b56dc36bcdd7cf5356b567a6a42dc20f53ee9b222b81fcb33bac1" Oct 30 19:15:02.622865 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.622844 5959 generic.go:356] "Generic (PLEG): container finished" podID="1bb68f2a-2ebc-4def-91d1-c8258f64dc84" containerID="c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9" exitCode=60 Oct 30 19:15:02.622953 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.622908 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" event={"ID":"1bb68f2a-2ebc-4def-91d1-c8258f64dc84","Type":"ContainerDied","Data":"c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9"} Oct 30 19:15:02.623202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.623186 5959 scope.go:122] "RemoveContainer" containerID="c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9" Oct 30 19:15:02.627147 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.627120 5959 generic.go:356] "Generic (PLEG): container finished" podID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerID="46a4fdc9c3ebffa18dd04d2e5cf5e620a5a9ed927f90fa3a062548d0128dd276" exitCode=60 Oct 30 19:15:02.627257 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.627233 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" event={"ID":"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4","Type":"ContainerDied","Data":"46a4fdc9c3ebffa18dd04d2e5cf5e620a5a9ed927f90fa3a062548d0128dd276"} Oct 30 19:15:02.627498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.627483 5959 scope.go:122] "RemoveContainer" containerID="46a4fdc9c3ebffa18dd04d2e5cf5e620a5a9ed927f90fa3a062548d0128dd276" Oct 30 19:15:02.629396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.629375 5959 generic.go:356] "Generic (PLEG): container finished" podID="8c0c1081-4209-4c4b-912c-e1defc785003" containerID="7872c6d490c9ae56ff561a185002e475ce98a195585074f5b61668cfd4a71273" exitCode=60 Oct 30 19:15:02.629684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.629530 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" event={"ID":"8c0c1081-4209-4c4b-912c-e1defc785003","Type":"ContainerDied","Data":"7872c6d490c9ae56ff561a185002e475ce98a195585074f5b61668cfd4a71273"} Oct 30 19:15:02.629865 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.629848 5959 scope.go:122] "RemoveContainer" containerID="7872c6d490c9ae56ff561a185002e475ce98a195585074f5b61668cfd4a71273" Oct 30 19:15:02.909716 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.909680 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4"] Oct 30 19:15:02.940535 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:02.940500 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" podUID="1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:03.651088 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:03.651039 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" event={"ID":"ef922752-ebe2-4428-81b5-95e146bff23f","Type":"ContainerStarted","Data":"8f29254cdcbc7c4c95628a0a5f3706d7d6ccea6540f754210454d766a3666ba7"} Oct 30 19:15:03.653758 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:03.653723 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" event={"ID":"1bb68f2a-2ebc-4def-91d1-c8258f64dc84","Type":"ContainerStarted","Data":"e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8"} Oct 30 19:15:03.653911 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:03.653838 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" podUID="1bb68f2a-2ebc-4def-91d1-c8258f64dc84" containerName="cleanup" containerID="cri-o://e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8" gracePeriod=30 Oct 30 19:15:03.659911 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:03.659876 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" event={"ID":"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4","Type":"ContainerStarted","Data":"40fd0e11c06bd0e5c6776ab9d1898891d43c3e8a2fae79836608aef2b0cd8d43"} Oct 30 19:15:03.672256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:03.670948 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" event={"ID":"8c0c1081-4209-4c4b-912c-e1defc785003","Type":"ContainerStarted","Data":"7f572f1bca3243a88892b014980b595f5f0031ad6023a477de2167a243847baf"} Oct 30 19:15:04.563622 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.563570 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp"] Oct 30 19:15:04.684123 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.684075 5959 generic.go:356] "Generic (PLEG): container finished" podID="ef922752-ebe2-4428-81b5-95e146bff23f" containerID="8f29254cdcbc7c4c95628a0a5f3706d7d6ccea6540f754210454d766a3666ba7" exitCode=60 Oct 30 19:15:04.684635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.684154 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" event={"ID":"ef922752-ebe2-4428-81b5-95e146bff23f","Type":"ContainerDied","Data":"8f29254cdcbc7c4c95628a0a5f3706d7d6ccea6540f754210454d766a3666ba7"} Oct 30 19:15:04.684635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.684220 5959 scope.go:122] "RemoveContainer" containerID="077695192f5b56dc36bcdd7cf5356b567a6a42dc20f53ee9b222b81fcb33bac1" Oct 30 19:15:04.684776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.684668 5959 scope.go:122] "RemoveContainer" containerID="8f29254cdcbc7c4c95628a0a5f3706d7d6ccea6540f754210454d766a3666ba7" Oct 30 19:15:04.685202 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:04.685140 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk_odh-ai-gateway-infra(ef922752-ebe2-4428-81b5-95e146bff23f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" Oct 30 19:15:04.686952 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.686929 5959 generic.go:356] "Generic (PLEG): container finished" podID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerID="40fd0e11c06bd0e5c6776ab9d1898891d43c3e8a2fae79836608aef2b0cd8d43" exitCode=60 Oct 30 19:15:04.687057 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.687015 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" event={"ID":"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4","Type":"ContainerDied","Data":"40fd0e11c06bd0e5c6776ab9d1898891d43c3e8a2fae79836608aef2b0cd8d43"} Oct 30 19:15:04.687386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.687363 5959 scope.go:122] "RemoveContainer" containerID="40fd0e11c06bd0e5c6776ab9d1898891d43c3e8a2fae79836608aef2b0cd8d43" Oct 30 19:15:04.687847 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:04.687820 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-rhkfr_odh-ai-gateway-infra(a61ed8c9-8ea9-4af5-a148-51a1b3b790e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" Oct 30 19:15:04.689130 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.689104 5959 generic.go:356] "Generic (PLEG): container finished" podID="8c0c1081-4209-4c4b-912c-e1defc785003" containerID="7f572f1bca3243a88892b014980b595f5f0031ad6023a477de2167a243847baf" exitCode=60 Oct 30 19:15:04.689637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.689135 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" event={"ID":"8c0c1081-4209-4c4b-912c-e1defc785003","Type":"ContainerDied","Data":"7f572f1bca3243a88892b014980b595f5f0031ad6023a477de2167a243847baf"} Oct 30 19:15:04.690363 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.690032 5959 scope.go:122] "RemoveContainer" containerID="7f572f1bca3243a88892b014980b595f5f0031ad6023a477de2167a243847baf" Oct 30 19:15:04.690819 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:04.690797 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q_odh-ai-gateway-infra(8c0c1081-4209-4c4b-912c-e1defc785003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" Oct 30 19:15:04.701132 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.701104 5959 scope.go:122] "RemoveContainer" containerID="46a4fdc9c3ebffa18dd04d2e5cf5e620a5a9ed927f90fa3a062548d0128dd276" Oct 30 19:15:04.763628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:04.759595 5959 scope.go:122] "RemoveContainer" containerID="7872c6d490c9ae56ff561a185002e475ce98a195585074f5b61668cfd4a71273" Oct 30 19:15:05.437397 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.437357 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g"] Oct 30 19:15:05.443642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.443620 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.452849 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.452823 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g"] Oct 30 19:15:05.457395 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.457175 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:05.606207 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.606168 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.606364 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.606293 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.606430 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.606361 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6qht\" (UniqueName: \"kubernetes.io/projected/bc349fb7-9184-4c24-a1c4-771fe430e078-kube-api-access-f6qht\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.705653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.705535 5959 scope.go:122] "RemoveContainer" containerID="40fd0e11c06bd0e5c6776ab9d1898891d43c3e8a2fae79836608aef2b0cd8d43" Oct 30 19:15:05.706081 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:05.706052 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-rhkfr_odh-ai-gateway-infra(a61ed8c9-8ea9-4af5-a148-51a1b3b790e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" Oct 30 19:15:05.707088 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.707063 5959 scope.go:122] "RemoveContainer" containerID="7f572f1bca3243a88892b014980b595f5f0031ad6023a477de2167a243847baf" Oct 30 19:15:05.707382 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:05.707358 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q_odh-ai-gateway-infra(8c0c1081-4209-4c4b-912c-e1defc785003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" Oct 30 19:15:05.708354 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.708327 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.708462 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.708406 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.708504 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.708461 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6qht\" (UniqueName: \"kubernetes.io/projected/bc349fb7-9184-4c24-a1c4-771fe430e078-kube-api-access-f6qht\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.709019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.708998 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.709464 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.709449 5959 scope.go:122] "RemoveContainer" containerID="8f29254cdcbc7c4c95628a0a5f3706d7d6ccea6540f754210454d766a3666ba7" Oct 30 19:15:05.709716 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:05.709698 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk_odh-ai-gateway-infra(ef922752-ebe2-4428-81b5-95e146bff23f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" Oct 30 19:15:05.711272 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.711253 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.717438 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.717417 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6qht\" (UniqueName: \"kubernetes.io/projected/bc349fb7-9184-4c24-a1c4-771fe430e078-kube-api-access-f6qht\") pod \"maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:05.792927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:05.792891 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:06.086813 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:06.086783 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g"] Oct 30 19:15:06.086973 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:15:06.086948 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc349fb7_9184_4c24_a1c4_771fe430e078.slice/crio-a838a481d7b5f9008b2262b6315988a8c19b5bb4e7ff0ec45c3f1afa045b3563 WatchSource:0}: Error finding container a838a481d7b5f9008b2262b6315988a8c19b5bb4e7ff0ec45c3f1afa045b3563: Status 404 returned error can't find the container with id a838a481d7b5f9008b2262b6315988a8c19b5bb4e7ff0ec45c3f1afa045b3563 Oct 30 19:15:06.720856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:06.720762 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" event={"ID":"bc349fb7-9184-4c24-a1c4-771fe430e078","Type":"ContainerStarted","Data":"3e18be774845d0e6acf5f2faf6d5e902486570069f6a5a705a541277317cc193"} Oct 30 19:15:06.720856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:06.720845 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" event={"ID":"bc349fb7-9184-4c24-a1c4-771fe430e078","Type":"ContainerStarted","Data":"a838a481d7b5f9008b2262b6315988a8c19b5bb4e7ff0ec45c3f1afa045b3563"} Oct 30 19:15:06.741197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:06.741147 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" podStartSLOduration=1.741133337 podStartE2EDuration="1.741133337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:06.738913921 +0000 UTC m=+2287.067325306" watchObservedRunningTime="2026-10-30 19:15:06.741133337 +0000 UTC m=+2287.069544722" Oct 30 19:15:07.731102 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:07.731065 5959 generic.go:356] "Generic (PLEG): container finished" podID="bc349fb7-9184-4c24-a1c4-771fe430e078" containerID="3e18be774845d0e6acf5f2faf6d5e902486570069f6a5a705a541277317cc193" exitCode=0 Oct 30 19:15:07.731507 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:07.731144 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" event={"ID":"bc349fb7-9184-4c24-a1c4-771fe430e078","Type":"ContainerDied","Data":"3e18be774845d0e6acf5f2faf6d5e902486570069f6a5a705a541277317cc193"} Oct 30 19:15:07.837621 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:07.837566 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm"] Oct 30 19:15:07.837817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:07.837677 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h"] Oct 30 19:15:07.837817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:07.837712 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72"] Oct 30 19:15:07.837955 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:07.837925 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" containerID="cri-o://61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1" gracePeriod=30 Oct 30 19:15:07.838292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:07.838260 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" containerID="cri-o://2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb" gracePeriod=30 Oct 30 19:15:07.838502 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:07.838462 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" containerID="cri-o://f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16" gracePeriod=30 Oct 30 19:15:08.914574 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:08.914547 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:08.982898 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:08.982857 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-openshift-ca\") pod \"bc349fb7-9184-4c24-a1c4-771fe430e078\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " Oct 30 19:15:08.983102 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:08.982917 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-xks-ca\") pod \"bc349fb7-9184-4c24-a1c4-771fe430e078\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " Oct 30 19:15:08.983102 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:08.983038 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc349fb7-9184-4c24-a1c4-771fe430e078-kube-api-access-f6qht\" (UniqueName: \"kubernetes.io/projected/bc349fb7-9184-4c24-a1c4-771fe430e078-kube-api-access-f6qht\") pod \"bc349fb7-9184-4c24-a1c4-771fe430e078\" (UID: \"bc349fb7-9184-4c24-a1c4-771fe430e078\") " Oct 30 19:15:08.983320 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:08.983291 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-openshift-ca" pod "bc349fb7-9184-4c24-a1c4-771fe430e078" (UID: "bc349fb7-9184-4c24-a1c4-771fe430e078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:08.983994 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:08.983970 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-openshift-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:08.986203 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:08.986152 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-xks-ca" pod "bc349fb7-9184-4c24-a1c4-771fe430e078" (UID: "bc349fb7-9184-4c24-a1c4-771fe430e078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:08.986320 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:08.986297 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc349fb7-9184-4c24-a1c4-771fe430e078-kube-api-access-f6qht" pod "bc349fb7-9184-4c24-a1c4-771fe430e078" (UID: "bc349fb7-9184-4c24-a1c4-771fe430e078"). InnerVolumeSpecName "kube-api-access-f6qht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:09.086050 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.086016 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc349fb7-9184-4c24-a1c4-771fe430e078-maas-api-xks-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:09.086050 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.086046 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6qht\" (UniqueName: \"kubernetes.io/projected/bc349fb7-9184-4c24-a1c4-771fe430e078-kube-api-access-f6qht\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:09.564869 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.564824 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" podUID="94520350-11fd-4ef1-9cd6-cf86f644cb74" containerName="payload-processing" containerID="cri-o://dcabfb551091b8029e048f69ddf0333e0f26bf99f66ffa58c198b1662e584bee" gracePeriod=25 Oct 30 19:15:09.782394 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.782358 5959 generic.go:356] "Generic (PLEG): container finished" podID="94520350-11fd-4ef1-9cd6-cf86f644cb74" containerID="dcabfb551091b8029e048f69ddf0333e0f26bf99f66ffa58c198b1662e584bee" exitCode=0 Oct 30 19:15:09.782588 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.782429 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" event={"ID":"94520350-11fd-4ef1-9cd6-cf86f644cb74","Type":"ContainerDied","Data":"dcabfb551091b8029e048f69ddf0333e0f26bf99f66ffa58c198b1662e584bee"} Oct 30 19:15:09.784223 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.784200 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" event={"ID":"bc349fb7-9184-4c24-a1c4-771fe430e078","Type":"ContainerDied","Data":"a838a481d7b5f9008b2262b6315988a8c19b5bb4e7ff0ec45c3f1afa045b3563"} Oct 30 19:15:09.784343 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.784243 5959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a838a481d7b5f9008b2262b6315988a8c19b5bb4e7ff0ec45c3f1afa045b3563" Oct 30 19:15:09.784343 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.784324 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g" Oct 30 19:15:09.835310 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.835280 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:15:09.900859 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.900829 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-kube-api-access-6wmx5\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-kube-api-access-6wmx5\") pod \"94520350-11fd-4ef1-9cd6-cf86f644cb74\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " Oct 30 19:15:09.901034 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.900877 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-provider-credentials\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-provider-credentials\") pod \"94520350-11fd-4ef1-9cd6-cf86f644cb74\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " Oct 30 19:15:09.901034 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.900948 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-routing-overlay-volume\") pod \"94520350-11fd-4ef1-9cd6-cf86f644cb74\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " Oct 30 19:15:09.901034 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.900975 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-plugins-config-volume\") pod \"94520350-11fd-4ef1-9cd6-cf86f644cb74\" (UID: \"94520350-11fd-4ef1-9cd6-cf86f644cb74\") " Oct 30 19:15:09.901398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.901366 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-routing-overlay-volume" pod "94520350-11fd-4ef1-9cd6-cf86f644cb74" (UID: "94520350-11fd-4ef1-9cd6-cf86f644cb74"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:09.901503 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.901409 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-plugins-config-volume" pod "94520350-11fd-4ef1-9cd6-cf86f644cb74" (UID: "94520350-11fd-4ef1-9cd6-cf86f644cb74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:09.901702 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.901680 5959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-routing-overlay-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:09.901808 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.901704 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94520350-11fd-4ef1-9cd6-cf86f644cb74-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:09.903572 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.903547 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-provider-credentials" pod "94520350-11fd-4ef1-9cd6-cf86f644cb74" (UID: "94520350-11fd-4ef1-9cd6-cf86f644cb74"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:09.903692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:09.903584 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-kube-api-access-6wmx5" pod "94520350-11fd-4ef1-9cd6-cf86f644cb74" (UID: "94520350-11fd-4ef1-9cd6-cf86f644cb74"). InnerVolumeSpecName "kube-api-access-6wmx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:10.004082 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:10.004045 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wmx5\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-kube-api-access-6wmx5\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:10.004082 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:10.004073 5959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94520350-11fd-4ef1-9cd6-cf86f644cb74-provider-credentials\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:10.793766 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:10.793735 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" Oct 30 19:15:10.793986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:10.793764 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp" event={"ID":"94520350-11fd-4ef1-9cd6-cf86f644cb74","Type":"ContainerDied","Data":"a179e050f963a1cb27fa6cd8824ee23efde139b83a2c126602b4490e14b2a0e7"} Oct 30 19:15:10.793986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:10.793816 5959 scope.go:122] "RemoveContainer" containerID="dcabfb551091b8029e048f69ddf0333e0f26bf99f66ffa58c198b1662e584bee" Oct 30 19:15:10.820201 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:10.820169 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp"] Oct 30 19:15:10.824304 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:10.824276 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d6f956/payload-processing-external-model-e2e-extmod-a-d6f956-7b9ct5zpp"] Oct 30 19:15:11.111728 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.107865 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5"] Oct 30 19:15:11.111728 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.108108 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" containerID="cri-o://6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3" gracePeriod=30 Oct 30 19:15:11.113436 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.113381 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh"] Oct 30 19:15:11.113701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.113655 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" containerID="cri-o://1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1" gracePeriod=30 Oct 30 19:15:11.118808 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.118783 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94"] Oct 30 19:15:11.118973 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.118955 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" containerID="cri-o://b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661" gracePeriod=30 Oct 30 19:15:11.572922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.572892 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5"] Oct 30 19:15:11.573759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.573740 5959 cpu_manager.go:414] "containerMap: removing container" podUID="94520350-11fd-4ef1-9cd6-cf86f644cb74" containerName="payload-processing" Oct 30 19:15:11.573759 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.573758 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94520350-11fd-4ef1-9cd6-cf86f644cb74" containerName="payload-processing" Oct 30 19:15:11.573905 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.573785 5959 cpu_manager.go:414] "containerMap: removing container" podUID="bc349fb7-9184-4c24-a1c4-771fe430e078" containerName="revoke-keys" Oct 30 19:15:11.573905 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.573790 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc349fb7-9184-4c24-a1c4-771fe430e078" containerName="revoke-keys" Oct 30 19:15:11.573980 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.573924 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="94520350-11fd-4ef1-9cd6-cf86f644cb74" containerName="payload-processing" Oct 30 19:15:11.573980 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.573939 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc349fb7-9184-4c24-a1c4-771fe430e078" containerName="revoke-keys" Oct 30 19:15:11.579313 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.579265 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.586126 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.586101 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5"] Oct 30 19:15:11.630036 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.629996 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:11.661098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.661061 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.661362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.661298 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.661362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.661335 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2cqn\" (UniqueName: \"kubernetes.io/projected/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-kube-api-access-f2cqn\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.762876 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.762841 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.763051 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.762990 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.763051 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.763013 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2cqn\" (UniqueName: \"kubernetes.io/projected/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-kube-api-access-f2cqn\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.763521 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.763490 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.765966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.765937 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.771530 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.771505 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2cqn\" (UniqueName: \"kubernetes.io/projected/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-kube-api-access-f2cqn\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:11.801775 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.801741 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94520350-11fd-4ef1-9cd6-cf86f644cb74" path="/var/lib/kubelet/pods/94520350-11fd-4ef1-9cd6-cf86f644cb74/volumes" Oct 30 19:15:11.854101 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.854017 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:15:11.854101 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.854069 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:15:11.898027 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:11.897995 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:12.193061 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:12.193037 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5"] Oct 30 19:15:12.193491 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:15:12.193469 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c4c0b9_5214_4046_8ce2_ef1da3161f17.slice/crio-213a22f951af80bec37588c9d35e56c942bea4e3d6a5cd028198093186e20763 WatchSource:0}: Error finding container 213a22f951af80bec37588c9d35e56c942bea4e3d6a5cd028198093186e20763: Status 404 returned error can't find the container with id 213a22f951af80bec37588c9d35e56c942bea4e3d6a5cd028198093186e20763 Oct 30 19:15:12.326810 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:12.326777 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:15:12.326975 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:12.326833 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:15:12.814920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:12.814822 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" event={"ID":"f1c4c0b9-5214-4046-8ce2-ef1da3161f17","Type":"ContainerStarted","Data":"0b8fa52cc6ad8b63ef3b14f5183cf8b9485beadc393ea0a777f0e16c82aaf974"} Oct 30 19:15:12.814920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:12.814884 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" event={"ID":"f1c4c0b9-5214-4046-8ce2-ef1da3161f17","Type":"ContainerStarted","Data":"213a22f951af80bec37588c9d35e56c942bea4e3d6a5cd028198093186e20763"} Oct 30 19:15:12.838568 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:12.838511 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" podStartSLOduration=1.838496756 podStartE2EDuration="1.838496756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:12.833621696 +0000 UTC m=+2293.162033078" watchObservedRunningTime="2026-10-30 19:15:12.838496756 +0000 UTC m=+2293.166908140" Oct 30 19:15:12.940660 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:12.940586 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" podUID="1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:13.160425 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.160391 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:15:13.168879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.168847 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:15:13.256663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.256630 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:15:13.298225 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298189 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-token\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-token\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.298411 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298294 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-credential-socket\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.298411 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298357 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-socket\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.298522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298426 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-kube-api-access-ftw6q\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-kube-api-access-ftw6q\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.298522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298474 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-kube-api-access-n6qgc\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-kube-api-access-n6qgc\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.298650 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298582 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-certs\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.298730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298647 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-data\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.298730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298690 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-envoy\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.298836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298773 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-podinfo\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.298836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298803 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istiod-ca-cert\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.298942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298833 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-socket\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.298942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.298881 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istiod-ca-cert\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.299074 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.299058 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-envoy\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.299128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.299114 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-certs\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.299209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.299195 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-token\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-token\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.299268 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.299250 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-credential-socket\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.299320 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.299300 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-data\") pod \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\" (UID: \"81fa02a0-22e8-4b53-924b-19e6cb605d5b\") " Oct 30 19:15:13.299373 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.299336 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-podinfo\") pod \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\" (UID: \"4cde4d2c-1782-41cc-9011-a7ac2ee0d079\") " Oct 30 19:15:13.300286 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.300262 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-data" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.300656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.300633 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-certs" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.300970 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.300954 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-credential-socket" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.302738 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.302712 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-kube-api-access-n6qgc" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "kube-api-access-n6qgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:13.305784 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.305758 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istiod-ca-cert" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:13.306655 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.306561 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-socket" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.306828 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.306804 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-socket" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.307433 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.306989 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-envoy" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.307433 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.307077 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istiod-ca-cert" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:13.307433 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.307252 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-credential-socket" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.307593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.307495 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-certs" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.308948 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.308473 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-data" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.309416 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.309394 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-token" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:13.310131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.310098 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-podinfo" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:13.310622 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.310568 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-envoy" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.310922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.310888 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-kube-api-access-ftw6q" pod "81fa02a0-22e8-4b53-924b-19e6cb605d5b" (UID: "81fa02a0-22e8-4b53-924b-19e6cb605d5b"). InnerVolumeSpecName "kube-api-access-ftw6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:13.311679 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.311654 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-token" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:13.313773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.313722 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-podinfo" pod "4cde4d2c-1782-41cc-9011-a7ac2ee0d079" (UID: "4cde4d2c-1782-41cc-9011-a7ac2ee0d079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:13.401362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401329 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-credential-socket\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.401557 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401443 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-token\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-token\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.401557 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401520 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-certs\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.401672 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401641 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-credential-socket" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.401712 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401679 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-data\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.401835 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401812 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istiod-ca-cert\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.401920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401851 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-certs" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.401920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401872 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-kube-api-access-5hpxn\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-kube-api-access-5hpxn\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.401920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401887 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-data" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.401920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401915 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-socket\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.402075 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.401950 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-podinfo\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.402075 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402063 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-envoy\") pod \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\" (UID: \"ab751835-0c83-473f-a8ff-b17e93a9a3a3\") " Oct 30 19:15:13.402179 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402096 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istiod-ca-cert" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:13.402239 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402180 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-socket" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.402811 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402788 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402814 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402831 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402844 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402857 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402872 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402884 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402899 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402911 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402924 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402936 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402949 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.402966 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402961 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402976 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.402989 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.403004 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.403018 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.403033 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.403047 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.403061 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.403078 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftw6q\" (UniqueName: \"kubernetes.io/projected/81fa02a0-22e8-4b53-924b-19e6cb605d5b-kube-api-access-ftw6q\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.403092 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.403420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.403107 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6qgc\" (UniqueName: \"kubernetes.io/projected/4cde4d2c-1782-41cc-9011-a7ac2ee0d079-kube-api-access-n6qgc\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.404319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.404292 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-envoy" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:13.404425 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.404296 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-podinfo" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:13.404555 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.404535 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-token" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:13.404641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.404572 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-kube-api-access-5hpxn" pod "ab751835-0c83-473f-a8ff-b17e93a9a3a3" (UID: "ab751835-0c83-473f-a8ff-b17e93a9a3a3"). InnerVolumeSpecName "kube-api-access-5hpxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:13.504467 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.504436 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.504467 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.504465 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.504724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.504476 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hpxn\" (UniqueName: \"kubernetes.io/projected/ab751835-0c83-473f-a8ff-b17e93a9a3a3-kube-api-access-5hpxn\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.504724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.504486 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab751835-0c83-473f-a8ff-b17e93a9a3a3-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:13.823790 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.823754 5959 generic.go:356] "Generic (PLEG): container finished" podID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerID="2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb" exitCode=0 Oct 30 19:15:13.823922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.823834 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" Oct 30 19:15:13.823922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.823834 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" event={"ID":"81fa02a0-22e8-4b53-924b-19e6cb605d5b","Type":"ContainerDied","Data":"2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb"} Oct 30 19:15:13.823922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.823890 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72" event={"ID":"81fa02a0-22e8-4b53-924b-19e6cb605d5b","Type":"ContainerDied","Data":"5e20afe1d01572ca35783b693b860ae2bebf4a9a9b04eea4cf2633eff46f2a18"} Oct 30 19:15:13.823922 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.823912 5959 scope.go:122] "RemoveContainer" containerID="2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb" Oct 30 19:15:13.825723 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.825660 5959 generic.go:356] "Generic (PLEG): container finished" podID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerID="f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16" exitCode=0 Oct 30 19:15:13.825861 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.825754 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" Oct 30 19:15:13.825861 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.825786 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" event={"ID":"4cde4d2c-1782-41cc-9011-a7ac2ee0d079","Type":"ContainerDied","Data":"f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16"} Oct 30 19:15:13.825861 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.825842 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h" event={"ID":"4cde4d2c-1782-41cc-9011-a7ac2ee0d079","Type":"ContainerDied","Data":"1e9578103612d90d703c89efe27f2912e9f0ce9d26aacd68b2867b0892d9920b"} Oct 30 19:15:13.827766 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.827744 5959 generic.go:356] "Generic (PLEG): container finished" podID="f1c4c0b9-5214-4046-8ce2-ef1da3161f17" containerID="0b8fa52cc6ad8b63ef3b14f5183cf8b9485beadc393ea0a777f0e16c82aaf974" exitCode=0 Oct 30 19:15:13.827887 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.827837 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" event={"ID":"f1c4c0b9-5214-4046-8ce2-ef1da3161f17","Type":"ContainerDied","Data":"0b8fa52cc6ad8b63ef3b14f5183cf8b9485beadc393ea0a777f0e16c82aaf974"} Oct 30 19:15:13.829589 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.829558 5959 generic.go:356] "Generic (PLEG): container finished" podID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerID="61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1" exitCode=0 Oct 30 19:15:13.829730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.829697 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" Oct 30 19:15:13.829730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.829699 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" event={"ID":"ab751835-0c83-473f-a8ff-b17e93a9a3a3","Type":"ContainerDied","Data":"61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1"} Oct 30 19:15:13.829856 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.829747 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm" event={"ID":"ab751835-0c83-473f-a8ff-b17e93a9a3a3","Type":"ContainerDied","Data":"0669874a33b840182c0f9584a8892e9fae33a98d586edb3ee88f303ab483fe55"} Oct 30 19:15:13.837188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.837168 5959 scope.go:122] "RemoveContainer" containerID="2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb" Oct 30 19:15:13.837464 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:13.837444 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb\": container with ID starting with 2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb not found: ID does not exist" containerID="2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb" Oct 30 19:15:13.837521 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.837472 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb"} err="failed to get container status \"2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb\": rpc error: code = NotFound desc = could not find container \"2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb\": container with ID starting with 2567db88c770969024fd42c4d96cda7c76b7492f1d6fb3fb4a0f04bde92684eb not found: ID does not exist" Oct 30 19:15:13.837521 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.837487 5959 scope.go:122] "RemoveContainer" containerID="f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16" Oct 30 19:15:13.850195 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.850175 5959 scope.go:122] "RemoveContainer" containerID="f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16" Oct 30 19:15:13.850577 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:13.850544 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16\": container with ID starting with f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16 not found: ID does not exist" containerID="f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16" Oct 30 19:15:13.850729 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.850586 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16"} err="failed to get container status \"f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16\": rpc error: code = NotFound desc = could not find container \"f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16\": container with ID starting with f8069ee45d4c530480a77325cf04cfc22be2c4ff96c5a57b6637f333693f6d16 not found: ID does not exist" Oct 30 19:15:13.850729 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.850649 5959 scope.go:122] "RemoveContainer" containerID="61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1" Oct 30 19:15:13.862736 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.862705 5959 scope.go:122] "RemoveContainer" containerID="61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1" Oct 30 19:15:13.863092 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:13.863060 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1\": container with ID starting with 61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1 not found: ID does not exist" containerID="61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1" Oct 30 19:15:13.863178 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.863101 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1"} err="failed to get container status \"61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1\": rpc error: code = NotFound desc = could not find container \"61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1\": container with ID starting with 61f1c8c749fd33329396fc80f203e2a08d108aeb39e79000514b367b2c7b16a1 not found: ID does not exist" Oct 30 19:15:13.876754 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.876723 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm"] Oct 30 19:15:13.879518 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.879494 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-vftmm"] Oct 30 19:15:13.886842 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.886820 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72"] Oct 30 19:15:13.893474 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.893449 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-twb72"] Oct 30 19:15:13.904756 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.904732 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h"] Oct 30 19:15:13.908522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:13.908502 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-c4g4h"] Oct 30 19:15:15.020217 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.020192 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:15.135904 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.135777 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-xks-ca\") pod \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " Oct 30 19:15:15.136146 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.135996 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-openshift-ca\") pod \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " Oct 30 19:15:15.136146 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.136092 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-kube-api-access-f2cqn\" (UniqueName: \"kubernetes.io/projected/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-kube-api-access-f2cqn\") pod \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\" (UID: \"f1c4c0b9-5214-4046-8ce2-ef1da3161f17\") " Oct 30 19:15:15.137394 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.137353 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-openshift-ca" pod "f1c4c0b9-5214-4046-8ce2-ef1da3161f17" (UID: "f1c4c0b9-5214-4046-8ce2-ef1da3161f17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:15.138645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.138589 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-xks-ca" pod "f1c4c0b9-5214-4046-8ce2-ef1da3161f17" (UID: "f1c4c0b9-5214-4046-8ce2-ef1da3161f17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:15.138761 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.138717 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-kube-api-access-f2cqn" pod "f1c4c0b9-5214-4046-8ce2-ef1da3161f17" (UID: "f1c4c0b9-5214-4046-8ce2-ef1da3161f17"). InnerVolumeSpecName "kube-api-access-f2cqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:15.238342 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.238303 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-openshift-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:15.238342 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.238333 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2cqn\" (UniqueName: \"kubernetes.io/projected/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-kube-api-access-f2cqn\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:15.238342 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.238349 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c4c0b9-5214-4046-8ce2-ef1da3161f17-maas-api-xks-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:15.650023 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.649985 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj"] Oct 30 19:15:15.831339 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.831301 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" path="/var/lib/kubelet/pods/4cde4d2c-1782-41cc-9011-a7ac2ee0d079/volumes" Oct 30 19:15:15.832450 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.832424 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" path="/var/lib/kubelet/pods/81fa02a0-22e8-4b53-924b-19e6cb605d5b/volumes" Oct 30 19:15:15.833209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.833191 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" path="/var/lib/kubelet/pods/ab751835-0c83-473f-a8ff-b17e93a9a3a3/volumes" Oct 30 19:15:15.854482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.853109 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" event={"ID":"f1c4c0b9-5214-4046-8ce2-ef1da3161f17","Type":"ContainerDied","Data":"213a22f951af80bec37588c9d35e56c942bea4e3d6a5cd028198093186e20763"} Oct 30 19:15:15.854482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.853177 5959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="213a22f951af80bec37588c9d35e56c942bea4e3d6a5cd028198093186e20763" Oct 30 19:15:15.854482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:15.853250 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5" Oct 30 19:15:16.625694 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.625308 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:15:16.700572 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.699567 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:15:16.717789 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.717759 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.731714 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a43000-522d-4ea3-9d38-4ceaa5789aea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a43000-522d-4ea3-9d38-4ceaa5789aea-istiod-ca-cert\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.731760 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-socket\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.731880 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-kube-api-access-p67s7\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-kube-api-access-p67s7\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.731979 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-credential-socket\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.732030 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-certs\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.732059 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-token\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-token\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.732102 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-data\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.732198 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-podinfo\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.733669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.732292 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-envoy\") pod \"48a43000-522d-4ea3-9d38-4ceaa5789aea\" (UID: \"48a43000-522d-4ea3-9d38-4ceaa5789aea\") " Oct 30 19:15:16.734470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.733747 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-certs" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.734470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.734351 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a43000-522d-4ea3-9d38-4ceaa5789aea-istiod-ca-cert" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:16.734584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.734575 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-data" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.734680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.734645 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-socket" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.741302 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.734823 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-credential-socket" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.750099 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.750058 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-envoy" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.751429 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.751404 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-podinfo" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:16.751579 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.751557 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-kube-api-access-p67s7" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "kube-api-access-p67s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.751662 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.751596 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-token" pod "48a43000-522d-4ea3-9d38-4ceaa5789aea" (UID: "48a43000-522d-4ea3-9d38-4ceaa5789aea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.789736 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.789701 5959 scope.go:122] "RemoveContainer" containerID="8f29254cdcbc7c4c95628a0a5f3706d7d6ccea6540f754210454d766a3666ba7" Oct 30 19:15:16.834554 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.834517 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-envoy\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.834772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.834586 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-envoy\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.834772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.834674 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/359be520-f69e-4d59-8a67-55e6a11236a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359be520-f69e-4d59-8a67-55e6a11236a5-istio-podinfo\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.834772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.834717 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-kube-api-access-p6bdl\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-kube-api-access-p6bdl\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.834772 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.834761 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-socket\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.835001 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.834801 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-kube-api-access-75s8w\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-kube-api-access-75s8w\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.835186 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835081 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359be520-f69e-4d59-8a67-55e6a11236a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359be520-f69e-4d59-8a67-55e6a11236a5-istiod-ca-cert\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.835186 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835144 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-istio-token\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-istio-token\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.835186 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835179 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istiod-ca-cert\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.835356 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835209 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-data\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.835356 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835245 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-data\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.835356 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835345 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-credential-socket\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.835576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835384 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-certs\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.835576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835437 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-socket\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.835576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835479 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-credential-socket\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.835576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835542 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-token\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-token\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.835870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835628 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-podinfo\") pod \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\" (UID: \"78abee8b-e5f9-4ff4-be79-ca766fbb26c0\") " Oct 30 19:15:16.835870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835633 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359be520-f69e-4d59-8a67-55e6a11236a5-istiod-ca-cert" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:16.835870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835657 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-certs\") pod \"359be520-f69e-4d59-8a67-55e6a11236a5\" (UID: \"359be520-f69e-4d59-8a67-55e6a11236a5\") " Oct 30 19:15:16.836070 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.835910 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-socket" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.836619 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.836421 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-data" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.836619 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.836512 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-certs" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.836946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.836763 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-certs" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.836946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.836792 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istiod-ca-cert" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:16.836946 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.836838 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-socket" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.837282 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837037 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-data" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.837282 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837113 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-credential-socket" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.837398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837379 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837399 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837413 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359be520-f69e-4d59-8a67-55e6a11236a5-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837428 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a43000-522d-4ea3-9d38-4ceaa5789aea-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837442 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837455 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837470 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837483 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837498 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837512 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67s7\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-kube-api-access-p67s7\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837528 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837539 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837552 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837565 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837580 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837594 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837627 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.837686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837641 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a43000-522d-4ea3-9d38-4ceaa5789aea-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.838342 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.837706 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-credential-socket" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.840192 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.840165 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-envoy" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.840416 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.840384 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-envoy" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:16.840507 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.840428 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-token" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.840507 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.840456 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-kube-api-access-75s8w" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "kube-api-access-75s8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.840658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.840596 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-podinfo" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:16.840873 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.840841 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/359be520-f69e-4d59-8a67-55e6a11236a5-istio-podinfo" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:16.841366 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.841341 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-kube-api-access-p6bdl" pod "78abee8b-e5f9-4ff4-be79-ca766fbb26c0" (UID: "78abee8b-e5f9-4ff4-be79-ca766fbb26c0"). InnerVolumeSpecName "kube-api-access-p6bdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.841917 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.841894 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-istio-token" pod "359be520-f69e-4d59-8a67-55e6a11236a5" (UID: "359be520-f69e-4d59-8a67-55e6a11236a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.863786 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.863756 5959 generic.go:356] "Generic (PLEG): container finished" podID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerID="b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661" exitCode=0 Oct 30 19:15:16.863962 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.863814 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" Oct 30 19:15:16.863962 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.863848 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" event={"ID":"78abee8b-e5f9-4ff4-be79-ca766fbb26c0","Type":"ContainerDied","Data":"b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661"} Oct 30 19:15:16.863962 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.863912 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94" event={"ID":"78abee8b-e5f9-4ff4-be79-ca766fbb26c0","Type":"ContainerDied","Data":"86eb5b5707f14f8cc445a519627d9fcae7019c7170216afae7b1a639d7a1886b"} Oct 30 19:15:16.863962 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.863941 5959 scope.go:122] "RemoveContainer" containerID="b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661" Oct 30 19:15:16.865311 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.865290 5959 generic.go:356] "Generic (PLEG): container finished" podID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerID="1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1" exitCode=0 Oct 30 19:15:16.865448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.865341 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" Oct 30 19:15:16.865448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.865355 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" event={"ID":"48a43000-522d-4ea3-9d38-4ceaa5789aea","Type":"ContainerDied","Data":"1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1"} Oct 30 19:15:16.865448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.865419 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh" event={"ID":"48a43000-522d-4ea3-9d38-4ceaa5789aea","Type":"ContainerDied","Data":"b1e4f57379516917889e796fbad4b3170f6065b368daaa782eacdf49ca59eb44"} Oct 30 19:15:16.877412 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.877389 5959 scope.go:122] "RemoveContainer" containerID="b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661" Oct 30 19:15:16.878217 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:16.878193 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661\": container with ID starting with b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661 not found: ID does not exist" containerID="b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661" Oct 30 19:15:16.878308 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.878236 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661"} err="failed to get container status \"b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661\": rpc error: code = NotFound desc = could not find container \"b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661\": container with ID starting with b952aea5713f5013af33bd3a2989323c984c9712695bbacafcffb6c0b73f7661 not found: ID does not exist" Oct 30 19:15:16.878308 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.878261 5959 scope.go:122] "RemoveContainer" containerID="1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1" Oct 30 19:15:16.881043 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.879399 5959 generic.go:356] "Generic (PLEG): container finished" podID="359be520-f69e-4d59-8a67-55e6a11236a5" containerID="6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3" exitCode=0 Oct 30 19:15:16.881043 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.879633 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" event={"ID":"359be520-f69e-4d59-8a67-55e6a11236a5","Type":"ContainerDied","Data":"6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3"} Oct 30 19:15:16.881043 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.879709 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" event={"ID":"359be520-f69e-4d59-8a67-55e6a11236a5","Type":"ContainerDied","Data":"53a544ade5c9330720e9e33dbf4cd1d1538bb8035916e9cbc719a72621a1a549"} Oct 30 19:15:16.881043 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.879838 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5" Oct 30 19:15:16.904536 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.904506 5959 scope.go:122] "RemoveContainer" containerID="1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1" Oct 30 19:15:16.905071 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:16.905022 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1\": container with ID starting with 1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1 not found: ID does not exist" containerID="1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1" Oct 30 19:15:16.905205 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.905179 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1"} err="failed to get container status \"1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1\": rpc error: code = NotFound desc = could not find container \"1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1\": container with ID starting with 1ed03a561833a50be4d8185934634ae7029590d68b408fab810c47e0ada7a4b1 not found: ID does not exist" Oct 30 19:15:16.905310 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.905298 5959 scope.go:122] "RemoveContainer" containerID="6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3" Oct 30 19:15:16.915229 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.912713 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh"] Oct 30 19:15:16.923918 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.923889 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-cp6fh"] Oct 30 19:15:16.936382 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.936349 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94"] Oct 30 19:15:16.939016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.938986 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.939016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.939018 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.939234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.939032 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.939234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.939044 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.939234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.939057 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359be520-f69e-4d59-8a67-55e6a11236a5-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.939234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.939069 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359be520-f69e-4d59-8a67-55e6a11236a5-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.939234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.939083 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6bdl\" (UniqueName: \"kubernetes.io/projected/78abee8b-e5f9-4ff4-be79-ca766fbb26c0-kube-api-access-p6bdl\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.939234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.939097 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75s8w\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-kube-api-access-75s8w\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.939234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.939109 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359be520-f69e-4d59-8a67-55e6a11236a5-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.940940 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.940914 5959 scope.go:122] "RemoveContainer" containerID="6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3" Oct 30 19:15:16.941535 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:16.941508 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3\": container with ID starting with 6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3 not found: ID does not exist" containerID="6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3" Oct 30 19:15:16.941693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.941547 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3"} err="failed to get container status \"6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3\": rpc error: code = NotFound desc = could not find container \"6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3\": container with ID starting with 6dfe28cbbda9fd18fd685a79ad84e9f9c46bc4493c9f752926d12d11307edcf3 not found: ID does not exist" Oct 30 19:15:16.942804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.942784 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-85m94"] Oct 30 19:15:16.956334 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.956271 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5"] Oct 30 19:15:16.967735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:16.967699 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-lh7n5"] Oct 30 19:15:17.790680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.790648 5959 scope.go:122] "RemoveContainer" containerID="40fd0e11c06bd0e5c6776ab9d1898891d43c3e8a2fae79836608aef2b0cd8d43" Oct 30 19:15:17.791193 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.790770 5959 scope.go:122] "RemoveContainer" containerID="7f572f1bca3243a88892b014980b595f5f0031ad6023a477de2167a243847baf" Oct 30 19:15:17.817173 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.817139 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" path="/var/lib/kubelet/pods/359be520-f69e-4d59-8a67-55e6a11236a5/volumes" Oct 30 19:15:17.818660 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.817975 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" path="/var/lib/kubelet/pods/48a43000-522d-4ea3-9d38-4ceaa5789aea/volumes" Oct 30 19:15:17.818831 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.818696 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" path="/var/lib/kubelet/pods/78abee8b-e5f9-4ff4-be79-ca766fbb26c0/volumes" Oct 30 19:15:17.895788 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.895755 5959 generic.go:356] "Generic (PLEG): container finished" podID="ef922752-ebe2-4428-81b5-95e146bff23f" containerID="30960e63e77e00f221540505b4a4a75c6a57ae4677a94ee222f580891d43dde0" exitCode=60 Oct 30 19:15:17.895939 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.895826 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" event={"ID":"ef922752-ebe2-4428-81b5-95e146bff23f","Type":"ContainerDied","Data":"30960e63e77e00f221540505b4a4a75c6a57ae4677a94ee222f580891d43dde0"} Oct 30 19:15:17.895939 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.895887 5959 scope.go:122] "RemoveContainer" containerID="8f29254cdcbc7c4c95628a0a5f3706d7d6ccea6540f754210454d766a3666ba7" Oct 30 19:15:17.896334 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:17.896318 5959 scope.go:122] "RemoveContainer" containerID="30960e63e77e00f221540505b4a4a75c6a57ae4677a94ee222f580891d43dde0" Oct 30 19:15:17.896888 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:17.896750 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk_odh-ai-gateway-infra(ef922752-ebe2-4428-81b5-95e146bff23f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" Oct 30 19:15:18.444951 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:18.444914 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:15:19.004812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.002429 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk"] Oct 30 19:15:19.004812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.002501 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" event={"ID":"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4","Type":"ContainerStarted","Data":"35425eb990c9e0eabef03864bfc40c2a0f503803bd43d9ee9faefa64607e629e"} Oct 30 19:15:19.007915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.007833 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" event={"ID":"8c0c1081-4209-4c4b-912c-e1defc785003","Type":"ContainerStarted","Data":"d40975f7b653f57a91f0b78f236d4b482eb704f0223c7d56d8302a135f7573f8"} Oct 30 19:15:19.187214 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.187187 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:19.278531 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.278437 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef922752-ebe2-4428-81b5-95e146bff23f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef922752-ebe2-4428-81b5-95e146bff23f-maas-api-service-ca\") pod \"ef922752-ebe2-4428-81b5-95e146bff23f\" (UID: \"ef922752-ebe2-4428-81b5-95e146bff23f\") " Oct 30 19:15:19.278776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.278754 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef922752-ebe2-4428-81b5-95e146bff23f-kube-api-access-wl5t6\" (UniqueName: \"kubernetes.io/projected/ef922752-ebe2-4428-81b5-95e146bff23f-kube-api-access-wl5t6\") pod \"ef922752-ebe2-4428-81b5-95e146bff23f\" (UID: \"ef922752-ebe2-4428-81b5-95e146bff23f\") " Oct 30 19:15:19.278901 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.278882 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef922752-ebe2-4428-81b5-95e146bff23f-maas-api-service-ca" pod "ef922752-ebe2-4428-81b5-95e146bff23f" (UID: "ef922752-ebe2-4428-81b5-95e146bff23f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:19.279448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.279433 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef922752-ebe2-4428-81b5-95e146bff23f-maas-api-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:19.281045 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.281025 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef922752-ebe2-4428-81b5-95e146bff23f-kube-api-access-wl5t6" pod "ef922752-ebe2-4428-81b5-95e146bff23f" (UID: "ef922752-ebe2-4428-81b5-95e146bff23f"). InnerVolumeSpecName "kube-api-access-wl5t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:19.380781 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:19.380749 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl5t6\" (UniqueName: \"kubernetes.io/projected/ef922752-ebe2-4428-81b5-95e146bff23f-kube-api-access-wl5t6\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:20.017773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.017738 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" Oct 30 19:15:20.018267 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.017771 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk" event={"ID":"ef922752-ebe2-4428-81b5-95e146bff23f","Type":"ContainerDied","Data":"cf775a138adafd4fd9bf5f4e039e7ea119e3fec4050f79422019fbca5531ecc7"} Oct 30 19:15:20.018267 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.017844 5959 scope.go:122] "RemoveContainer" containerID="30960e63e77e00f221540505b4a4a75c6a57ae4677a94ee222f580891d43dde0" Oct 30 19:15:20.020277 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.020158 5959 generic.go:356] "Generic (PLEG): container finished" podID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerID="35425eb990c9e0eabef03864bfc40c2a0f503803bd43d9ee9faefa64607e629e" exitCode=60 Oct 30 19:15:20.020401 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.020274 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" event={"ID":"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4","Type":"ContainerDied","Data":"35425eb990c9e0eabef03864bfc40c2a0f503803bd43d9ee9faefa64607e629e"} Oct 30 19:15:20.020810 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.020789 5959 scope.go:122] "RemoveContainer" containerID="35425eb990c9e0eabef03864bfc40c2a0f503803bd43d9ee9faefa64607e629e" Oct 30 19:15:20.021216 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:20.021195 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-rhkfr_odh-ai-gateway-infra(a61ed8c9-8ea9-4af5-a148-51a1b3b790e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" Oct 30 19:15:20.022418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.022401 5959 generic.go:356] "Generic (PLEG): container finished" podID="8c0c1081-4209-4c4b-912c-e1defc785003" containerID="d40975f7b653f57a91f0b78f236d4b482eb704f0223c7d56d8302a135f7573f8" exitCode=60 Oct 30 19:15:20.022522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.022477 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" event={"ID":"8c0c1081-4209-4c4b-912c-e1defc785003","Type":"ContainerDied","Data":"d40975f7b653f57a91f0b78f236d4b482eb704f0223c7d56d8302a135f7573f8"} Oct 30 19:15:20.022850 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.022837 5959 scope.go:122] "RemoveContainer" containerID="d40975f7b653f57a91f0b78f236d4b482eb704f0223c7d56d8302a135f7573f8" Oct 30 19:15:20.023114 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:20.023098 5959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q_odh-ai-gateway-infra(8c0c1081-4209-4c4b-912c-e1defc785003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" Oct 30 19:15:20.029341 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.029321 5959 scope.go:122] "RemoveContainer" containerID="40fd0e11c06bd0e5c6776ab9d1898891d43c3e8a2fae79836608aef2b0cd8d43" Oct 30 19:15:20.035110 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.035087 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr"] Oct 30 19:15:20.039533 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.039513 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk"] Oct 30 19:15:20.045322 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.045176 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889795-jrczk"] Oct 30 19:15:20.045480 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.045447 5959 scope.go:122] "RemoveContainer" containerID="7f572f1bca3243a88892b014980b595f5f0031ad6023a477de2167a243847baf" Oct 30 19:15:20.071176 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.071136 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q"] Oct 30 19:15:20.651326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.651266 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" podUID="f126e5dc-9d83-4e52-8c42-ca8390a07330" containerName="payload-pre-processing" containerID="cri-o://c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69" gracePeriod=25 Oct 30 19:15:20.881035 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.881005 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:15:20.911645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.911591 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-plugins-config-volume\") pod \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\" (UID: \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\") " Oct 30 19:15:20.911956 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.911934 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-plugins-config-volume" pod "1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" (UID: "1b7b40fc-9360-4a4b-baa0-02fcb9063ac0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.911956 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.911941 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-kube-api-access-fmf8h\" (UniqueName: \"kubernetes.io/projected/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-kube-api-access-fmf8h\") pod \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\" (UID: \"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0\") " Oct 30 19:15:20.913120 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.913093 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:20.914524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:20.914494 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-kube-api-access-fmf8h" pod "1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" (UID: "1b7b40fc-9360-4a4b-baa0-02fcb9063ac0"). InnerVolumeSpecName "kube-api-access-fmf8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:21.014323 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.014291 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmf8h\" (UniqueName: \"kubernetes.io/projected/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0-kube-api-access-fmf8h\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.034010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.033973 5959 generic.go:356] "Generic (PLEG): container finished" podID="1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" containerID="d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6" exitCode=0 Oct 30 19:15:21.034010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.034010 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" Oct 30 19:15:21.034527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.034038 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" event={"ID":"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0","Type":"ContainerDied","Data":"d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6"} Oct 30 19:15:21.034527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.034091 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7" event={"ID":"1b7b40fc-9360-4a4b-baa0-02fcb9063ac0","Type":"ContainerDied","Data":"c9606b95952f586cb7fb21b01e04f3b99e338a9d7aa131e56b42b17257f0856e"} Oct 30 19:15:21.034527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.034120 5959 scope.go:122] "RemoveContainer" containerID="d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6" Oct 30 19:15:21.051289 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.051246 5959 scope.go:122] "RemoveContainer" containerID="d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6" Oct 30 19:15:21.051659 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:21.051633 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6\": container with ID starting with d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6 not found: ID does not exist" containerID="d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6" Oct 30 19:15:21.051740 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.051673 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6"} err="failed to get container status \"d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6\": rpc error: code = NotFound desc = could not find container \"d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6\": container with ID starting with d854bd9653fae25a71ce07bdd46cef48c8fbec7a97b30b35bd0dbc5f88f399e6 not found: ID does not exist" Oct 30 19:15:21.089647 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.088094 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7"] Oct 30 19:15:21.096642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.095679 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e67d95-8b4987555-hgkf7"] Oct 30 19:15:21.294685 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.287571 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:21.320981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.320943 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-maas-api-service-ca\") pod \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\" (UID: \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\") " Oct 30 19:15:21.321264 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.321243 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-kube-api-access-8mwqz\" (UniqueName: \"kubernetes.io/projected/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-kube-api-access-8mwqz\") pod \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\" (UID: \"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4\") " Oct 30 19:15:21.325995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.322717 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-maas-api-service-ca" pod "a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" (UID: "a61ed8c9-8ea9-4af5-a148-51a1b3b790e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:21.336335 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.336295 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-kube-api-access-8mwqz" pod "a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" (UID: "a61ed8c9-8ea9-4af5-a148-51a1b3b790e4"). InnerVolumeSpecName "kube-api-access-8mwqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:21.377026 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.376212 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:21.424885 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.424847 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0c1081-4209-4c4b-912c-e1defc785003-kube-api-access-752h5\" (UniqueName: \"kubernetes.io/projected/8c0c1081-4209-4c4b-912c-e1defc785003-kube-api-access-752h5\") pod \"8c0c1081-4209-4c4b-912c-e1defc785003\" (UID: \"8c0c1081-4209-4c4b-912c-e1defc785003\") " Oct 30 19:15:21.425174 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.425014 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0c1081-4209-4c4b-912c-e1defc785003-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0c1081-4209-4c4b-912c-e1defc785003-maas-api-service-ca\") pod \"8c0c1081-4209-4c4b-912c-e1defc785003\" (UID: \"8c0c1081-4209-4c4b-912c-e1defc785003\") " Oct 30 19:15:21.425717 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.425692 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mwqz\" (UniqueName: \"kubernetes.io/projected/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-kube-api-access-8mwqz\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.425870 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.425720 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4-maas-api-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.426469 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.426438 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0c1081-4209-4c4b-912c-e1defc785003-maas-api-service-ca" pod "8c0c1081-4209-4c4b-912c-e1defc785003" (UID: "8c0c1081-4209-4c4b-912c-e1defc785003"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:21.429284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.429255 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0c1081-4209-4c4b-912c-e1defc785003-kube-api-access-752h5" pod "8c0c1081-4209-4c4b-912c-e1defc785003" (UID: "8c0c1081-4209-4c4b-912c-e1defc785003"). InnerVolumeSpecName "kube-api-access-752h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:21.526927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.526891 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-752h5\" (UniqueName: \"kubernetes.io/projected/8c0c1081-4209-4c4b-912c-e1defc785003-kube-api-access-752h5\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.526927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.526926 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0c1081-4209-4c4b-912c-e1defc785003-maas-api-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.809930 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.809888 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" path="/var/lib/kubelet/pods/1b7b40fc-9360-4a4b-baa0-02fcb9063ac0/volumes" Oct 30 19:15:21.810583 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:21.810559 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" path="/var/lib/kubelet/pods/ef922752-ebe2-4428-81b5-95e146bff23f/volumes" Oct 30 19:15:22.200509 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.200472 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" event={"ID":"a61ed8c9-8ea9-4af5-a148-51a1b3b790e4","Type":"ContainerDied","Data":"30cadd5887abb65d8e85197128a60ecba084be35c613a728e18bba161d0eaac5"} Oct 30 19:15:22.200993 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.200553 5959 scope.go:122] "RemoveContainer" containerID="35425eb990c9e0eabef03864bfc40c2a0f503803bd43d9ee9faefa64607e629e" Oct 30 19:15:22.200993 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.200698 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr" Oct 30 19:15:22.204078 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.204043 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" event={"ID":"8c0c1081-4209-4c4b-912c-e1defc785003","Type":"ContainerDied","Data":"1c7ccaa838e730d6a15a2cd4f63ec9ddd9bd4f89718cbcfbd36ebbe137b4256e"} Oct 30 19:15:22.204253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.204156 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q" Oct 30 19:15:22.219101 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.219078 5959 scope.go:122] "RemoveContainer" containerID="d40975f7b653f57a91f0b78f236d4b482eb704f0223c7d56d8302a135f7573f8" Oct 30 19:15:22.230951 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.230923 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr"] Oct 30 19:15:22.236500 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.236469 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-rhkfr"] Oct 30 19:15:22.252711 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.252675 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q"] Oct 30 19:15:22.259413 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:22.259386 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28c1e9-29889795-fpn4q"] Oct 30 19:15:23.518500 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:23.518463 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l"] Oct 30 19:15:23.519024 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:23.518767 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" podUID="036f65e0-fe98-4601-b469-3c033588c81f" containerName="maas-api" containerID="cri-o://b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab" gracePeriod=30 Oct 30 19:15:23.803992 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:23.803898 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" path="/var/lib/kubelet/pods/8c0c1081-4209-4c4b-912c-e1defc785003/volumes" Oct 30 19:15:23.804621 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:23.804581 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" path="/var/lib/kubelet/pods/a61ed8c9-8ea9-4af5-a148-51a1b3b790e4/volumes" Oct 30 19:15:24.000750 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:23.997733 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" podUID="f126e5dc-9d83-4e52-8c42-ca8390a07330" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:24.021651 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.019833 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:15:24.077211 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.076644 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-metrics-certs\") pod \"036f65e0-fe98-4601-b469-3c033588c81f\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " Oct 30 19:15:24.077398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.077274 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls\") pod \"036f65e0-fe98-4601-b469-3c033588c81f\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " Oct 30 19:15:24.077728 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.077703 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036f65e0-fe98-4601-b469-3c033588c81f-kube-api-access-t4lr9\" (UniqueName: \"kubernetes.io/projected/036f65e0-fe98-4601-b469-3c033588c81f-kube-api-access-t4lr9\") pod \"036f65e0-fe98-4601-b469-3c033588c81f\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " Oct 30 19:15:24.078139 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.078116 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036f65e0-fe98-4601-b469-3c033588c81f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/036f65e0-fe98-4601-b469-3c033588c81f-openshift-service-ca\") pod \"036f65e0-fe98-4601-b469-3c033588c81f\" (UID: \"036f65e0-fe98-4601-b469-3c033588c81f\") " Oct 30 19:15:24.079309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.079281 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036f65e0-fe98-4601-b469-3c033588c81f-openshift-service-ca" pod "036f65e0-fe98-4601-b469-3c033588c81f" (UID: "036f65e0-fe98-4601-b469-3c033588c81f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:24.091207 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.091129 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls" pod "036f65e0-fe98-4601-b469-3c033588c81f" (UID: "036f65e0-fe98-4601-b469-3c033588c81f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:24.093834 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.093786 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036f65e0-fe98-4601-b469-3c033588c81f-kube-api-access-t4lr9" pod "036f65e0-fe98-4601-b469-3c033588c81f" (UID: "036f65e0-fe98-4601-b469-3c033588c81f"). InnerVolumeSpecName "kube-api-access-t4lr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:24.118553 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.118518 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-metrics-certs" pod "036f65e0-fe98-4601-b469-3c033588c81f" (UID: "036f65e0-fe98-4601-b469-3c033588c81f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:24.181081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.181045 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:24.181081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.181078 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/036f65e0-fe98-4601-b469-3c033588c81f-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:24.181081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.181090 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4lr9\" (UniqueName: \"kubernetes.io/projected/036f65e0-fe98-4601-b469-3c033588c81f-kube-api-access-t4lr9\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:24.181325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.181100 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/036f65e0-fe98-4601-b469-3c033588c81f-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:24.226345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.226312 5959 generic.go:356] "Generic (PLEG): container finished" podID="036f65e0-fe98-4601-b469-3c033588c81f" containerID="b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab" exitCode=0 Oct 30 19:15:24.226546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.226359 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" Oct 30 19:15:24.226546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.226407 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" event={"ID":"036f65e0-fe98-4601-b469-3c033588c81f","Type":"ContainerDied","Data":"b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab"} Oct 30 19:15:24.226546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.226460 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l" event={"ID":"036f65e0-fe98-4601-b469-3c033588c81f","Type":"ContainerDied","Data":"7facc0145e873d090efed1cc8d611dee9d4f486b789abed888b75d373d915840"} Oct 30 19:15:24.226546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.226487 5959 scope.go:122] "RemoveContainer" containerID="b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab" Oct 30 19:15:24.242446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.242422 5959 scope.go:122] "RemoveContainer" containerID="b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab" Oct 30 19:15:24.242870 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:24.242841 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab\": container with ID starting with b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab not found: ID does not exist" containerID="b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab" Oct 30 19:15:24.242980 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.242883 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab"} err="failed to get container status \"b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab\": rpc error: code = NotFound desc = could not find container \"b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab\": container with ID starting with b7b913d72db24f267762a71e3a2d58e3da2c61b018d9705f7242105fb56eacab not found: ID does not exist" Oct 30 19:15:24.255933 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.255463 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l"] Oct 30 19:15:24.259327 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.259296 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28c1e9-864874d84d-t9v4l"] Oct 30 19:15:24.340675 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.340533 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5"] Oct 30 19:15:24.344171 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:24.344143 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ef6b56-qtmm5"] Oct 30 19:15:25.800152 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:25.800116 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036f65e0-fe98-4601-b469-3c033588c81f" path="/var/lib/kubelet/pods/036f65e0-fe98-4601-b469-3c033588c81f/volumes" Oct 30 19:15:25.800673 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:25.800655 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c4c0b9-5214-4046-8ce2-ef1da3161f17" path="/var/lib/kubelet/pods/f1c4c0b9-5214-4046-8ce2-ef1da3161f17/volumes" Oct 30 19:15:26.452821 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:26.452791 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp"] Oct 30 19:15:27.784007 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:27.783967 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:15:29.975642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:29.975589 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg"] Oct 30 19:15:29.985863 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:29.985829 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" podUID="89938e5d-7f44-484a-ab58-355bc498e717" containerName="maas-api" containerID="cri-o://f23c664469fc019859231feb87220a10ec2d1f2cc28a0ea691ea324403548306" gracePeriod=30 Oct 30 19:15:30.399157 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.397668 5959 generic.go:356] "Generic (PLEG): container finished" podID="89938e5d-7f44-484a-ab58-355bc498e717" containerID="f23c664469fc019859231feb87220a10ec2d1f2cc28a0ea691ea324403548306" exitCode=0 Oct 30 19:15:30.399157 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.397830 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" event={"ID":"89938e5d-7f44-484a-ab58-355bc498e717","Type":"ContainerDied","Data":"f23c664469fc019859231feb87220a10ec2d1f2cc28a0ea691ea324403548306"} Oct 30 19:15:30.615284 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.615257 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:15:30.683476 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.683386 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-metrics-certs\") pod \"89938e5d-7f44-484a-ab58-355bc498e717\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " Oct 30 19:15:30.684045 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.684017 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89938e5d-7f44-484a-ab58-355bc498e717-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89938e5d-7f44-484a-ab58-355bc498e717-openshift-service-ca\") pod \"89938e5d-7f44-484a-ab58-355bc498e717\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " Oct 30 19:15:30.684288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.684260 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-maas-api-tls\") pod \"89938e5d-7f44-484a-ab58-355bc498e717\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " Oct 30 19:15:30.684419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.684359 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89938e5d-7f44-484a-ab58-355bc498e717-kube-api-access-d2wk9\" (UniqueName: \"kubernetes.io/projected/89938e5d-7f44-484a-ab58-355bc498e717-kube-api-access-d2wk9\") pod \"89938e5d-7f44-484a-ab58-355bc498e717\" (UID: \"89938e5d-7f44-484a-ab58-355bc498e717\") " Oct 30 19:15:30.685476 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.685451 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g"] Oct 30 19:15:30.687127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.687099 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-metrics-certs" pod "89938e5d-7f44-484a-ab58-355bc498e717" (UID: "89938e5d-7f44-484a-ab58-355bc498e717"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:30.690306 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.690277 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89938e5d-7f44-484a-ab58-355bc498e717-kube-api-access-d2wk9" pod "89938e5d-7f44-484a-ab58-355bc498e717" (UID: "89938e5d-7f44-484a-ab58-355bc498e717"). InnerVolumeSpecName "kube-api-access-d2wk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:30.690420 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.685475 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89938e5d-7f44-484a-ab58-355bc498e717-openshift-service-ca" pod "89938e5d-7f44-484a-ab58-355bc498e717" (UID: "89938e5d-7f44-484a-ab58-355bc498e717"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:30.715677 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.715596 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d6f956-n755g"] Oct 30 19:15:30.716537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.716508 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-maas-api-tls" pod "89938e5d-7f44-484a-ab58-355bc498e717" (UID: "89938e5d-7f44-484a-ab58-355bc498e717"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:30.786439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.786400 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:30.786720 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.786457 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wk9\" (UniqueName: \"kubernetes.io/projected/89938e5d-7f44-484a-ab58-355bc498e717-kube-api-access-d2wk9\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:30.786720 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.786469 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89938e5d-7f44-484a-ab58-355bc498e717-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:30.786720 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:30.786479 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89938e5d-7f44-484a-ab58-355bc498e717-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:31.406786 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:31.406750 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" event={"ID":"89938e5d-7f44-484a-ab58-355bc498e717","Type":"ContainerDied","Data":"f9c716e0239050db553f62e46247975c0e662ba068154686415db7b65aa467b1"} Oct 30 19:15:31.406786 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:31.406774 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg" Oct 30 19:15:31.407457 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:31.406818 5959 scope.go:122] "RemoveContainer" containerID="f23c664469fc019859231feb87220a10ec2d1f2cc28a0ea691ea324403548306" Oct 30 19:15:31.431030 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:31.430989 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg"] Oct 30 19:15:31.436446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:31.436414 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6f956-749dd8d7dc-pqscg"] Oct 30 19:15:31.453638 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:31.453556 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" podUID="21ebf898-38d1-4786-8ebc-d5e53a52961f" containerName="payload-processing" containerID="cri-o://7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214" gracePeriod=25 Oct 30 19:15:31.800059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:31.799974 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89938e5d-7f44-484a-ab58-355bc498e717" path="/var/lib/kubelet/pods/89938e5d-7f44-484a-ab58-355bc498e717/volumes" Oct 30 19:15:31.800504 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:31.800490 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc349fb7-9184-4c24-a1c4-771fe430e078" path="/var/lib/kubelet/pods/bc349fb7-9184-4c24-a1c4-771fe430e078/volumes" Oct 30 19:15:32.747944 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:32.747898 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" podUID="21ebf898-38d1-4786-8ebc-d5e53a52961f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:33.879807 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:33.879781 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:33.955101 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:33.955008 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" podUID="f126e5dc-9d83-4e52-8c42-ca8390a07330" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:34.016913 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.016878 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-kube-api-access-7t2h9\" (UniqueName: \"kubernetes.io/projected/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-kube-api-access-7t2h9\") pod \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\" (UID: \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\") " Oct 30 19:15:34.017129 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.017069 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-maas-api-service-ca\") pod \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\" (UID: \"1bb68f2a-2ebc-4def-91d1-c8258f64dc84\") " Oct 30 19:15:34.017419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.017391 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-maas-api-service-ca" pod "1bb68f2a-2ebc-4def-91d1-c8258f64dc84" (UID: "1bb68f2a-2ebc-4def-91d1-c8258f64dc84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:34.017766 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.017748 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-maas-api-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:34.019209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.019186 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-kube-api-access-7t2h9" pod "1bb68f2a-2ebc-4def-91d1-c8258f64dc84" (UID: "1bb68f2a-2ebc-4def-91d1-c8258f64dc84"). InnerVolumeSpecName "kube-api-access-7t2h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:34.119005 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.118964 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2h9\" (UniqueName: \"kubernetes.io/projected/1bb68f2a-2ebc-4def-91d1-c8258f64dc84-kube-api-access-7t2h9\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:34.438165 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.438128 5959 generic.go:356] "Generic (PLEG): container finished" podID="1bb68f2a-2ebc-4def-91d1-c8258f64dc84" containerID="e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8" exitCode=137 Oct 30 19:15:34.438396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.438209 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" Oct 30 19:15:34.438396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.438248 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" event={"ID":"1bb68f2a-2ebc-4def-91d1-c8258f64dc84","Type":"ContainerDied","Data":"e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8"} Oct 30 19:15:34.438396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.438336 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4" event={"ID":"1bb68f2a-2ebc-4def-91d1-c8258f64dc84","Type":"ContainerDied","Data":"a7ca780d71c2a8185a58f67836ba610173b7e178d5fd0b0f7f47ae086bb4eacd"} Oct 30 19:15:34.438396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.438363 5959 scope.go:122] "RemoveContainer" containerID="e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8" Oct 30 19:15:34.449971 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.449951 5959 scope.go:122] "RemoveContainer" containerID="c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9" Oct 30 19:15:34.460442 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.460424 5959 scope.go:122] "RemoveContainer" containerID="e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8" Oct 30 19:15:34.460790 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:34.460769 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8\": container with ID starting with e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8 not found: ID does not exist" containerID="e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8" Oct 30 19:15:34.460862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.460799 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8"} err="failed to get container status \"e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8\": rpc error: code = NotFound desc = could not find container \"e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8\": container with ID starting with e289238bc56d4c8e4bd2908f303c0549610daf8e9d0e77f3ef345a493cd49ba8 not found: ID does not exist" Oct 30 19:15:34.460862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.460815 5959 scope.go:122] "RemoveContainer" containerID="c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9" Oct 30 19:15:34.461081 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:34.461062 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9\": container with ID starting with c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9 not found: ID does not exist" containerID="c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9" Oct 30 19:15:34.461129 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.461088 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9"} err="failed to get container status \"c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9\": rpc error: code = NotFound desc = could not find container \"c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9\": container with ID starting with c3e14725931de46e8846d31a063af8a88feace252ce07b7d782103dfd87097a9 not found: ID does not exist" Oct 30 19:15:34.463124 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.463102 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4"] Oct 30 19:15:34.469228 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:34.469206 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e67d95-29889795-8rrz4"] Oct 30 19:15:35.799927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:35.799897 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb68f2a-2ebc-4def-91d1-c8258f64dc84" path="/var/lib/kubelet/pods/1bb68f2a-2ebc-4def-91d1-c8258f64dc84/volumes" Oct 30 19:15:38.054671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:38.054590 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5"] Oct 30 19:15:38.055218 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:38.054905 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" containerID="cri-o://8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97" gracePeriod=30 Oct 30 19:15:38.063158 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:38.063125 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp"] Oct 30 19:15:38.063418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:38.063372 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" containerID="cri-o://03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2" gracePeriod=30 Oct 30 19:15:38.088538 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:38.088504 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr"] Oct 30 19:15:38.088844 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:38.088810 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" containerID="cri-o://05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5" gracePeriod=30 Oct 30 19:15:39.921691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:39.921656 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:15:39.922165 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:39.921719 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:15:40.938822 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:40.938757 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:15:41.052707 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.052645 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f126e5dc-9d83-4e52-8c42-ca8390a07330-kube-api-access-5q49v\" (UniqueName: \"kubernetes.io/projected/f126e5dc-9d83-4e52-8c42-ca8390a07330-kube-api-access-5q49v\") pod \"f126e5dc-9d83-4e52-8c42-ca8390a07330\" (UID: \"f126e5dc-9d83-4e52-8c42-ca8390a07330\") " Oct 30 19:15:41.053092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.053068 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f126e5dc-9d83-4e52-8c42-ca8390a07330-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f126e5dc-9d83-4e52-8c42-ca8390a07330-plugins-config-volume\") pod \"f126e5dc-9d83-4e52-8c42-ca8390a07330\" (UID: \"f126e5dc-9d83-4e52-8c42-ca8390a07330\") " Oct 30 19:15:41.054461 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.053537 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f126e5dc-9d83-4e52-8c42-ca8390a07330-plugins-config-volume" pod "f126e5dc-9d83-4e52-8c42-ca8390a07330" (UID: "f126e5dc-9d83-4e52-8c42-ca8390a07330"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:41.064869 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.064826 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f126e5dc-9d83-4e52-8c42-ca8390a07330-kube-api-access-5q49v" pod "f126e5dc-9d83-4e52-8c42-ca8390a07330" (UID: "f126e5dc-9d83-4e52-8c42-ca8390a07330"). InnerVolumeSpecName "kube-api-access-5q49v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:41.156034 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.155997 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q49v\" (UniqueName: \"kubernetes.io/projected/f126e5dc-9d83-4e52-8c42-ca8390a07330-kube-api-access-5q49v\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:41.156034 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.156030 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f126e5dc-9d83-4e52-8c42-ca8390a07330-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:41.498096 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.498058 5959 generic.go:356] "Generic (PLEG): container finished" podID="f126e5dc-9d83-4e52-8c42-ca8390a07330" containerID="c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69" exitCode=0 Oct 30 19:15:41.498307 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.498105 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" event={"ID":"f126e5dc-9d83-4e52-8c42-ca8390a07330","Type":"ContainerDied","Data":"c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69"} Oct 30 19:15:41.498307 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.498153 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" event={"ID":"f126e5dc-9d83-4e52-8c42-ca8390a07330","Type":"ContainerDied","Data":"83a40c0ae4f76e965b725138b39c3f63c49e091c7b3fccbda72bb2c516e3a57c"} Oct 30 19:15:41.498307 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.498180 5959 scope.go:122] "RemoveContainer" containerID="c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69" Oct 30 19:15:41.498307 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.498108 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj" Oct 30 19:15:41.510474 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.510451 5959 scope.go:122] "RemoveContainer" containerID="c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69" Oct 30 19:15:41.510825 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:41.510805 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69\": container with ID starting with c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69 not found: ID does not exist" containerID="c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69" Oct 30 19:15:41.510917 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.510834 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69"} err="failed to get container status \"c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69\": rpc error: code = NotFound desc = could not find container \"c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69\": container with ID starting with c66018dae42dfd32cdac72132ae46efdbaa8a82e98e98d1cf268beb6c7089c69 not found: ID does not exist" Oct 30 19:15:41.523693 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.523665 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj"] Oct 30 19:15:41.531709 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.531589 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6f956-6cd456fccf-mtrqj"] Oct 30 19:15:41.798828 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:41.798747 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f126e5dc-9d83-4e52-8c42-ca8390a07330" path="/var/lib/kubelet/pods/f126e5dc-9d83-4e52-8c42-ca8390a07330/volumes" Oct 30 19:15:42.747315 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:42.747282 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" podUID="21ebf898-38d1-4786-8ebc-d5e53a52961f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:42.825883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:42.825845 5959 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:15:42.826070 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:42.825905 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:15:43.335188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.335149 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:15:43.383044 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.383021 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:15:43.467056 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.467020 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:15:43.499032 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499003 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-kube-api-access-bxt2x\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-kube-api-access-bxt2x\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.499230 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499045 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-token\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-token\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.499230 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499081 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-data\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.499230 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499112 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-certs\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.499230 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499184 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-token\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.499502 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499473 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-data" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.499732 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499498 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-certs" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.499732 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499689 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-credential-socket\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.499732 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499746 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-socket\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.500011 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499778 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-certs\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.500011 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499966 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-credential-socket" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.500011 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.499973 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-envoy\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.500189 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500077 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-podinfo\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.500189 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500154 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istiod-ca-cert\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.500311 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500187 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-credential-socket\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.500397 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500356 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-socket\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.500468 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500424 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istiod-ca-cert\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.500527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500498 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-data\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.500584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500539 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-kube-api-access-54xw2\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-kube-api-access-54xw2\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.500584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500569 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-podinfo\") pod \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\" (UID: \"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e\") " Oct 30 19:15:43.500718 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.500596 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-envoy\") pod \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\" (UID: \"c654a045-0394-4c1f-a8fe-a67f9e22ea2e\") " Oct 30 19:15:43.501507 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.501483 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.501642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.501513 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.501642 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.501528 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.501778 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.501760 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-kube-api-access-bxt2x" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "kube-api-access-bxt2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:43.501932 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.501915 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-socket" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.502155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502103 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-credential-socket" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.502155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502116 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-certs" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.502155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502182 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istiod-ca-cert" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:43.502465 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502378 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-token" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:43.502465 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502400 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istiod-ca-cert" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:43.502465 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502434 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-envoy" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.502692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502595 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-token" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:43.502692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502667 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-data" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.502817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.502712 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-socket" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.503940 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.503909 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-envoy" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.504035 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.504005 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-podinfo" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:43.504366 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.504343 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-kube-api-access-54xw2" pod "c654a045-0394-4c1f-a8fe-a67f9e22ea2e" (UID: "c654a045-0394-4c1f-a8fe-a67f9e22ea2e"). InnerVolumeSpecName "kube-api-access-54xw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:43.505034 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.505007 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-podinfo" pod "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" (UID: "3b7b9b00-3bc3-4ece-8926-41355d0a9b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:43.517700 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.517654 5959 generic.go:356] "Generic (PLEG): container finished" podID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerID="03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2" exitCode=0 Oct 30 19:15:43.517836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.517703 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" Oct 30 19:15:43.517836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.517734 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" event={"ID":"c654a045-0394-4c1f-a8fe-a67f9e22ea2e","Type":"ContainerDied","Data":"03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2"} Oct 30 19:15:43.517836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.517805 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp" event={"ID":"c654a045-0394-4c1f-a8fe-a67f9e22ea2e","Type":"ContainerDied","Data":"a56409a247799df403297cddcfa66cf8126d2539bb9f1edeeb0327a0500ae857"} Oct 30 19:15:43.518018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.517838 5959 scope.go:122] "RemoveContainer" containerID="03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2" Oct 30 19:15:43.520153 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.520128 5959 generic.go:356] "Generic (PLEG): container finished" podID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerID="05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5" exitCode=0 Oct 30 19:15:43.520379 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.520361 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" Oct 30 19:15:43.520518 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.520498 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" event={"ID":"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34","Type":"ContainerDied","Data":"05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5"} Oct 30 19:15:43.520586 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.520543 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr" event={"ID":"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34","Type":"ContainerDied","Data":"020f9eec948b57ac4020ee3a738f39159646e3e6d4ff2505ac7f028ec01f3914"} Oct 30 19:15:43.522463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.522440 5959 generic.go:356] "Generic (PLEG): container finished" podID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerID="8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97" exitCode=0 Oct 30 19:15:43.522582 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.522515 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" Oct 30 19:15:43.525668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.522475 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" event={"ID":"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e","Type":"ContainerDied","Data":"8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97"} Oct 30 19:15:43.525775 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.525703 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5" event={"ID":"3b7b9b00-3bc3-4ece-8926-41355d0a9b5e","Type":"ContainerDied","Data":"faf027692ebfeaf74a1a52cd325a23350df4d823cc59003775bb9841a32a00d3"} Oct 30 19:15:43.577520 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.577491 5959 scope.go:122] "RemoveContainer" containerID="03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2" Oct 30 19:15:43.577975 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:43.577949 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2\": container with ID starting with 03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2 not found: ID does not exist" containerID="03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2" Oct 30 19:15:43.578107 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.578080 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2"} err="failed to get container status \"03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2\": rpc error: code = NotFound desc = could not find container \"03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2\": container with ID starting with 03c00697e7795a3443c3492a547821b54e30039165351235b653863f7002b7e2 not found: ID does not exist" Oct 30 19:15:43.578202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.578111 5959 scope.go:122] "RemoveContainer" containerID="05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607054 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-kube-api-access-gzrbr\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-kube-api-access-gzrbr\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607137 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-data\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607259 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-certs\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607406 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-socket\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607469 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istiod-ca-cert\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607595 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-podinfo\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607712 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-credential-socket\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607765 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-envoy\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.607793 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-token\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-token\") pod \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\" (UID: \"5ceedfeb-9369-4ad4-bb4a-d23c669a6e34\") " Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608540 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608561 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608575 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608590 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54xw2\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-kube-api-access-54xw2\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608625 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608638 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608653 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxt2x\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-kube-api-access-bxt2x\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608667 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608681 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608694 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608707 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608720 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608733 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c654a045-0394-4c1f-a8fe-a67f9e22ea2e-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608745 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.630824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.608758 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.647206 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.647010 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-credential-socket" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.653629 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.647589 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-data" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.671717 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.665742 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-socket" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.671717 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.666006 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-certs" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.696032 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.688264 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istiod-ca-cert" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:43.697410 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.697376 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp"] Oct 30 19:15:43.703085 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.702265 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-envoy" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:43.703085 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.702504 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-podinfo" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:43.703908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.703854 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-token" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:43.707692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.707654 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-s86lp"] Oct 30 19:15:43.710981 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.710956 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.711127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.711111 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.711248 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.711234 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.711348 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.711333 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.711439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.711427 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.711527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.711516 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.711628 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.711598 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.711715 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.711704 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.712227 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.712187 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-kube-api-access-gzrbr" pod "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" (UID: "5ceedfeb-9369-4ad4-bb4a-d23c669a6e34"). InnerVolumeSpecName "kube-api-access-gzrbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:43.719712 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.718493 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t"] Oct 30 19:15:43.736575 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.733675 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9"] Oct 30 19:15:43.739155 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.738240 5959 scope.go:122] "RemoveContainer" containerID="05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5" Oct 30 19:15:43.750633 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.748619 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5"] Oct 30 19:15:43.752081 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:43.751585 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5\": container with ID starting with 05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5 not found: ID does not exist" containerID="05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5" Oct 30 19:15:43.752081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.751656 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5"} err="failed to get container status \"05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5\": rpc error: code = NotFound desc = could not find container \"05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5\": container with ID starting with 05ae90c6f5fe98c6126c7c50e1b8b020bea52b932189ccf292dce7877e3b72f5 not found: ID does not exist" Oct 30 19:15:43.752081 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.751691 5959 scope.go:122] "RemoveContainer" containerID="8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97" Oct 30 19:15:43.783727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.783679 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-pzhn5"] Oct 30 19:15:43.799306 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.799279 5959 scope.go:122] "RemoveContainer" containerID="8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97" Oct 30 19:15:43.799699 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:43.799673 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97\": container with ID starting with 8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97 not found: ID does not exist" containerID="8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97" Oct 30 19:15:43.799792 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.799709 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97"} err="failed to get container status \"8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97\": rpc error: code = NotFound desc = could not find container \"8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97\": container with ID starting with 8d969bc53a977a35ec69aa80e31ad3e8943321460a9420eef7f2a06e8ee9ac97 not found: ID does not exist" Oct 30 19:15:43.806806 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.806758 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" path="/var/lib/kubelet/pods/3b7b9b00-3bc3-4ece-8926-41355d0a9b5e/volumes" Oct 30 19:15:43.807568 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.807475 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" path="/var/lib/kubelet/pods/c654a045-0394-4c1f-a8fe-a67f9e22ea2e/volumes" Oct 30 19:15:43.813088 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.813061 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrbr\" (UniqueName: \"kubernetes.io/projected/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34-kube-api-access-gzrbr\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:43.840682 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.840646 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr"] Oct 30 19:15:43.846213 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:43.846183 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-n58tr"] Oct 30 19:15:45.798516 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:45.798483 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" path="/var/lib/kubelet/pods/5ceedfeb-9369-4ad4-bb4a-d23c669a6e34/volumes" Oct 30 19:15:48.765782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:48.765749 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n"] Oct 30 19:15:48.785410 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:48.785372 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj"] Oct 30 19:15:49.159867 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:49.159831 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp"] Oct 30 19:15:49.177720 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:49.177633 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7"] Oct 30 19:15:50.666808 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:50.666771 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:15:51.805009 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:51.804979 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:15:51.871997 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:51.871951 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ebf898-38d1-4786-8ebc-d5e53a52961f-kube-api-access-zttlp\" (UniqueName: \"kubernetes.io/projected/21ebf898-38d1-4786-8ebc-d5e53a52961f-kube-api-access-zttlp\") pod \"21ebf898-38d1-4786-8ebc-d5e53a52961f\" (UID: \"21ebf898-38d1-4786-8ebc-d5e53a52961f\") " Oct 30 19:15:51.872364 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:51.872343 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ebf898-38d1-4786-8ebc-d5e53a52961f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ebf898-38d1-4786-8ebc-d5e53a52961f-plugins-config-volume\") pod \"21ebf898-38d1-4786-8ebc-d5e53a52961f\" (UID: \"21ebf898-38d1-4786-8ebc-d5e53a52961f\") " Oct 30 19:15:51.872965 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:51.872935 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ebf898-38d1-4786-8ebc-d5e53a52961f-plugins-config-volume" pod "21ebf898-38d1-4786-8ebc-d5e53a52961f" (UID: "21ebf898-38d1-4786-8ebc-d5e53a52961f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:51.873132 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:51.873113 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ebf898-38d1-4786-8ebc-d5e53a52961f-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:51.874446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:51.874418 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ebf898-38d1-4786-8ebc-d5e53a52961f-kube-api-access-zttlp" pod "21ebf898-38d1-4786-8ebc-d5e53a52961f" (UID: "21ebf898-38d1-4786-8ebc-d5e53a52961f"). InnerVolumeSpecName "kube-api-access-zttlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:51.975052 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:51.975017 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttlp\" (UniqueName: \"kubernetes.io/projected/21ebf898-38d1-4786-8ebc-d5e53a52961f-kube-api-access-zttlp\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:52.639664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.639591 5959 generic.go:356] "Generic (PLEG): container finished" podID="21ebf898-38d1-4786-8ebc-d5e53a52961f" containerID="7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214" exitCode=0 Oct 30 19:15:52.639850 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.639665 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" Oct 30 19:15:52.639850 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.639681 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" event={"ID":"21ebf898-38d1-4786-8ebc-d5e53a52961f","Type":"ContainerDied","Data":"7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214"} Oct 30 19:15:52.639850 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.639764 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp" event={"ID":"21ebf898-38d1-4786-8ebc-d5e53a52961f","Type":"ContainerDied","Data":"182ee51656881b7212941e051ac45183e65a9278dfd9d57728c67db955d4af6c"} Oct 30 19:15:52.639850 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.639792 5959 scope.go:122] "RemoveContainer" containerID="7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214" Oct 30 19:15:52.651930 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.651907 5959 scope.go:122] "RemoveContainer" containerID="7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214" Oct 30 19:15:52.652320 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:52.652297 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214\": container with ID starting with 7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214 not found: ID does not exist" containerID="7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214" Oct 30 19:15:52.652404 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.652331 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214"} err="failed to get container status \"7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214\": rpc error: code = NotFound desc = could not find container \"7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214\": container with ID starting with 7e8ac170dce11e578cb632f5f29fa60ff5d091bf07410064cb9488ecb05f6214 not found: ID does not exist" Oct 30 19:15:52.666730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.666681 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp"] Oct 30 19:15:52.670254 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:52.670221 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-68b466-7db5d4cd-j47fp"] Oct 30 19:15:53.766701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:53.766660 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" podUID="74f87578-e8f2-4a8a-9e6b-0525fd6c45db" containerName="payload-processing" containerID="cri-o://fd9b34ac8a7794d89bd360cb45d41c0463e7322ba711e58f49f7fd2402e8291a" gracePeriod=25 Oct 30 19:15:53.786519 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:53.786474 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" podUID="f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" containerName="payload-pre-processing" containerID="cri-o://33eda19fdb078aec365aa12b580b264237cafd8922e2c9bc665b257923a73d41" gracePeriod=25 Oct 30 19:15:53.807241 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:53.807206 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ebf898-38d1-4786-8ebc-d5e53a52961f" path="/var/lib/kubelet/pods/21ebf898-38d1-4786-8ebc-d5e53a52961f/volumes" Oct 30 19:15:54.170689 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:54.165735 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" podUID="cff7c99c-0d72-4542-a2ef-28910e83d0bf" containerName="payload-processing" containerID="cri-o://d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead" gracePeriod=25 Oct 30 19:15:54.184815 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:54.184235 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" podUID="23bd45ad-4844-4771-ab04-be34f44f15f2" containerName="payload-pre-processing" containerID="cri-o://94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0" gracePeriod=25 Oct 30 19:15:55.520440 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:55.520396 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" podUID="f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:55.520924 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:55.520412 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" podUID="74f87578-e8f2-4a8a-9e6b-0525fd6c45db" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:57.795584 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:57.795549 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:15:58.207017 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.206950 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp"] Oct 30 19:15:58.208150 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.208117 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" podUID="a1c0c547-d980-42ee-8b82-411a2bf05e64" containerName="maas-api" containerID="cri-o://fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2" gracePeriod=30 Oct 30 19:15:58.658488 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.658457 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:15:58.694935 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.694900 5959 generic.go:356] "Generic (PLEG): container finished" podID="a1c0c547-d980-42ee-8b82-411a2bf05e64" containerID="fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2" exitCode=0 Oct 30 19:15:58.695135 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.694965 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" Oct 30 19:15:58.695135 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.694994 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" event={"ID":"a1c0c547-d980-42ee-8b82-411a2bf05e64","Type":"ContainerDied","Data":"fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2"} Oct 30 19:15:58.695135 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.695070 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp" event={"ID":"a1c0c547-d980-42ee-8b82-411a2bf05e64","Type":"ContainerDied","Data":"1bf213fbcc480cfb4dfc3beaf6268377b2e6174f5095f8052c96c7c7e502c7eb"} Oct 30 19:15:58.695135 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.695117 5959 scope.go:122] "RemoveContainer" containerID="fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2" Oct 30 19:15:58.707995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.707967 5959 scope.go:122] "RemoveContainer" containerID="fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2" Oct 30 19:15:58.708355 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:58.708333 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2\": container with ID starting with fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2 not found: ID does not exist" containerID="fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2" Oct 30 19:15:58.708486 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.708364 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2"} err="failed to get container status \"fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2\": rpc error: code = NotFound desc = could not find container \"fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2\": container with ID starting with fd17f6fa01cbcbdfa3c39fe3e0dc4067980b3452e70b352dba7e942b013e25f2 not found: ID does not exist" Oct 30 19:15:58.741285 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.741250 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls\") pod \"a1c0c547-d980-42ee-8b82-411a2bf05e64\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " Oct 30 19:15:58.741556 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.741344 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c0c547-d980-42ee-8b82-411a2bf05e64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c0c547-d980-42ee-8b82-411a2bf05e64-openshift-service-ca\") pod \"a1c0c547-d980-42ee-8b82-411a2bf05e64\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " Oct 30 19:15:58.741556 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.741525 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c0c547-d980-42ee-8b82-411a2bf05e64-kube-api-access-76c6k\" (UniqueName: \"kubernetes.io/projected/a1c0c547-d980-42ee-8b82-411a2bf05e64-kube-api-access-76c6k\") pod \"a1c0c547-d980-42ee-8b82-411a2bf05e64\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " Oct 30 19:15:58.741661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.741636 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-metrics-certs\") pod \"a1c0c547-d980-42ee-8b82-411a2bf05e64\" (UID: \"a1c0c547-d980-42ee-8b82-411a2bf05e64\") " Oct 30 19:15:58.741746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.741724 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c0c547-d980-42ee-8b82-411a2bf05e64-openshift-service-ca" pod "a1c0c547-d980-42ee-8b82-411a2bf05e64" (UID: "a1c0c547-d980-42ee-8b82-411a2bf05e64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:58.742548 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.742520 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1c0c547-d980-42ee-8b82-411a2bf05e64-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:58.743816 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.743785 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c0c547-d980-42ee-8b82-411a2bf05e64-kube-api-access-76c6k" pod "a1c0c547-d980-42ee-8b82-411a2bf05e64" (UID: "a1c0c547-d980-42ee-8b82-411a2bf05e64"). InnerVolumeSpecName "kube-api-access-76c6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:58.743932 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.743830 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls" pod "a1c0c547-d980-42ee-8b82-411a2bf05e64" (UID: "a1c0c547-d980-42ee-8b82-411a2bf05e64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:58.743932 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.743868 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-metrics-certs" pod "a1c0c547-d980-42ee-8b82-411a2bf05e64" (UID: "a1c0c547-d980-42ee-8b82-411a2bf05e64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:58.776223 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.776186 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" containerID="cri-o://36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb" gracePeriod=45 Oct 30 19:15:58.811266 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.811230 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" containerID="cri-o://5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949" gracePeriod=45 Oct 30 19:15:58.870689 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.863096 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76c6k\" (UniqueName: \"kubernetes.io/projected/a1c0c547-d980-42ee-8b82-411a2bf05e64-kube-api-access-76c6k\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:58.870689 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.863141 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:58.870689 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:58.863166 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1c0c547-d980-42ee-8b82-411a2bf05e64-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.034663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.033245 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp"] Oct 30 19:15:59.043579 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.040194 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a477d-8545544489-d82bp"] Oct 30 19:15:59.125444 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.125357 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:15:59.166563 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.166530 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:15:59.275023 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.274983 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-tmp-dir\") pod \"e300afa7-3bad-4988-a244-2c0dfb446993\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " Oct 30 19:15:59.275023 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275023 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-home\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-home\") pod \"e300afa7-3bad-4988-a244-2c0dfb446993\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " Oct 30 19:15:59.275272 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275071 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-dshm\") pod \"e300afa7-3bad-4988-a244-2c0dfb446993\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " Oct 30 19:15:59.275272 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275182 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-model-cache\") pod \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " Oct 30 19:15:59.275272 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275253 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-dshm\") pod \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " Oct 30 19:15:59.275460 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275265 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-tmp-dir" pod "e300afa7-3bad-4988-a244-2c0dfb446993" (UID: "e300afa7-3bad-4988-a244-2c0dfb446993"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:59.275460 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275355 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-home" pod "e300afa7-3bad-4988-a244-2c0dfb446993" (UID: "e300afa7-3bad-4988-a244-2c0dfb446993"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:59.275460 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275373 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-kube-api-access-2n6tt\" (UniqueName: \"kubernetes.io/projected/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-kube-api-access-2n6tt\") pod \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " Oct 30 19:15:59.275672 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275467 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-model-cache" pod "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" (UID: "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:59.275672 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275474 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e300afa7-3bad-4988-a244-2c0dfb446993-kube-api-access-tmhdk\" (UniqueName: \"kubernetes.io/projected/e300afa7-3bad-4988-a244-2c0dfb446993-kube-api-access-tmhdk\") pod \"e300afa7-3bad-4988-a244-2c0dfb446993\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " Oct 30 19:15:59.275672 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275573 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-model-cache\") pod \"e300afa7-3bad-4988-a244-2c0dfb446993\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " Oct 30 19:15:59.275672 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275635 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-home\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-home\") pod \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " Oct 30 19:15:59.276080 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275695 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tls-certs\") pod \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " Oct 30 19:15:59.276080 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275751 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e300afa7-3bad-4988-a244-2c0dfb446993-tls-certs\" (UniqueName: \"kubernetes.io/secret/e300afa7-3bad-4988-a244-2c0dfb446993-tls-certs\") pod \"e300afa7-3bad-4988-a244-2c0dfb446993\" (UID: \"e300afa7-3bad-4988-a244-2c0dfb446993\") " Oct 30 19:15:59.276080 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275794 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tmp-dir\") pod \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\" (UID: \"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2\") " Oct 30 19:15:59.276080 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.275855 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-model-cache" pod "e300afa7-3bad-4988-a244-2c0dfb446993" (UID: "e300afa7-3bad-4988-a244-2c0dfb446993"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:59.276345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.276080 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-home" pod "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" (UID: "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:59.276345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.276285 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tmp-dir" pod "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" (UID: "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:59.277072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.276883 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.277072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.276910 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.277072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.276928 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.277072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.276941 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.277072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.276953 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.277072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.276966 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.278325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.278290 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tls-certs" pod "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" (UID: "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:59.278472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.278448 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-dshm" pod "e300afa7-3bad-4988-a244-2c0dfb446993" (UID: "e300afa7-3bad-4988-a244-2c0dfb446993"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:59.278537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.278466 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e300afa7-3bad-4988-a244-2c0dfb446993-kube-api-access-tmhdk" pod "e300afa7-3bad-4988-a244-2c0dfb446993" (UID: "e300afa7-3bad-4988-a244-2c0dfb446993"). InnerVolumeSpecName "kube-api-access-tmhdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:59.278661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.278639 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e300afa7-3bad-4988-a244-2c0dfb446993-tls-certs" pod "e300afa7-3bad-4988-a244-2c0dfb446993" (UID: "e300afa7-3bad-4988-a244-2c0dfb446993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:59.278879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.278848 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-kube-api-access-2n6tt" pod "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" (UID: "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2"). InnerVolumeSpecName "kube-api-access-2n6tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:59.279485 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.279465 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-dshm" pod "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" (UID: "6cb88e37-7ad7-4e05-b2af-f2571e01a2d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:59.378432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.378337 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.378432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.378370 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e300afa7-3bad-4988-a244-2c0dfb446993-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.378432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.378378 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e300afa7-3bad-4988-a244-2c0dfb446993-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.378432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.378386 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.378432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.378397 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n6tt\" (UniqueName: \"kubernetes.io/projected/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2-kube-api-access-2n6tt\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.378432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.378409 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmhdk\" (UniqueName: \"kubernetes.io/projected/e300afa7-3bad-4988-a244-2c0dfb446993-kube-api-access-tmhdk\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.706653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.706596 5959 generic.go:356] "Generic (PLEG): container finished" podID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerID="36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb" exitCode=0 Oct 30 19:15:59.706862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.706658 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" event={"ID":"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2","Type":"ContainerDied","Data":"36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb"} Oct 30 19:15:59.706862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.706685 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" Oct 30 19:15:59.706862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.706710 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9" event={"ID":"6cb88e37-7ad7-4e05-b2af-f2571e01a2d2","Type":"ContainerDied","Data":"1837ac92b4d7cf795da2db388d990843b06b711117054971b1f33d4ad49818b4"} Oct 30 19:15:59.706862 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.706731 5959 scope.go:122] "RemoveContainer" containerID="36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb" Oct 30 19:15:59.708415 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.708387 5959 generic.go:356] "Generic (PLEG): container finished" podID="e300afa7-3bad-4988-a244-2c0dfb446993" containerID="5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949" exitCode=0 Oct 30 19:15:59.708717 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.708443 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" Oct 30 19:15:59.708717 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.708579 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" event={"ID":"e300afa7-3bad-4988-a244-2c0dfb446993","Type":"ContainerDied","Data":"5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949"} Oct 30 19:15:59.708717 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.708663 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t" event={"ID":"e300afa7-3bad-4988-a244-2c0dfb446993","Type":"ContainerDied","Data":"034e9e003c19cfcb0830804d57351137bc566aee41eeea3d0768695c45ad586e"} Oct 30 19:15:59.726844 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.726043 5959 scope.go:122] "RemoveContainer" containerID="36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb" Oct 30 19:15:59.726844 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:59.726650 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb\": container with ID starting with 36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb not found: ID does not exist" containerID="36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb" Oct 30 19:15:59.726844 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.726684 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb"} err="failed to get container status \"36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb\": rpc error: code = NotFound desc = could not find container \"36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb\": container with ID starting with 36fbdb15b472f8002d76bcf9db17e1a6f8821ea385df6b4734e7c1179a695dcb not found: ID does not exist" Oct 30 19:15:59.726844 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.726706 5959 scope.go:122] "RemoveContainer" containerID="5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949" Oct 30 19:15:59.744522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.744492 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9"] Oct 30 19:15:59.746741 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.746719 5959 scope.go:122] "RemoveContainer" containerID="5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949" Oct 30 19:15:59.747150 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:15:59.747123 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949\": container with ID starting with 5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949 not found: ID does not exist" containerID="5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949" Oct 30 19:15:59.747253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.747167 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949"} err="failed to get container status \"5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949\": rpc error: code = NotFound desc = could not find container \"5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949\": container with ID starting with 5d9f755e0cec33f1b238f0e21f6d75cc24ddf25de919518391a73cc5542af949 not found: ID does not exist" Oct 30 19:15:59.748141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.748116 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/simulator-w0-9de872-kserve-75bb5f55fb-4bln9"] Oct 30 19:15:59.761309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.761276 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t"] Oct 30 19:15:59.767776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.767746 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9de872/distinct-w0-9de872-kserve-5dc4b6c6c6-krf9t"] Oct 30 19:15:59.802106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.802072 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" path="/var/lib/kubelet/pods/6cb88e37-7ad7-4e05-b2af-f2571e01a2d2/volumes" Oct 30 19:15:59.802686 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.802669 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c0c547-d980-42ee-8b82-411a2bf05e64" path="/var/lib/kubelet/pods/a1c0c547-d980-42ee-8b82-411a2bf05e64/volumes" Oct 30 19:15:59.803099 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:15:59.803085 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" path="/var/lib/kubelet/pods/e300afa7-3bad-4988-a244-2c0dfb446993/volumes" Oct 30 19:16:00.003532 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:00.003443 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" podUID="23bd45ad-4844-4771-ab04-be34f44f15f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:00.003995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:00.003443 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" podUID="cff7c99c-0d72-4542-a2ef-28910e83d0bf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:00.369088 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:00.368996 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg"] Oct 30 19:16:00.369544 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:00.369514 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" containerID="cri-o://b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b" gracePeriod=30 Oct 30 19:16:05.554632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.554282 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" podUID="74f87578-e8f2-4a8a-9e6b-0525fd6c45db" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:05.557023 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.556992 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" podUID="f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:05.701714 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.701678 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:16:05.811958 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.811920 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-data\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.812183 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.811972 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-credential-socket\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.812183 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.812068 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-token\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-token\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.812183 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.812102 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-kube-api-access-4lppd\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-kube-api-access-4lppd\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.812183 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.812138 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-envoy\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.812419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.812268 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-podinfo\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.812419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.812344 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-socket\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.812419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.812406 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-certs\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.812580 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.812497 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istiod-ca-cert\") pod \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\" (UID: \"6bd33ce2-06d2-4680-af9a-58e00ea3ac03\") " Oct 30 19:16:05.828641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.825883 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istiod-ca-cert" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:05.828641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.826349 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-data" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:05.828641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.826840 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-credential-socket" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:05.877661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.877492 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-certs" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:05.877965 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.877919 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-socket" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:05.907637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.893947 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-envoy" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:05.907637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.896086 5959 generic.go:356] "Generic (PLEG): container finished" podID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerID="b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b" exitCode=0 Oct 30 19:16:05.907637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.896446 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" Oct 30 19:16:05.907637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.897894 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-kube-api-access-4lppd" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "kube-api-access-4lppd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:05.913892 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.913853 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" event={"ID":"6bd33ce2-06d2-4680-af9a-58e00ea3ac03","Type":"ContainerDied","Data":"b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b"} Oct 30 19:16:05.914136 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.913928 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg" event={"ID":"6bd33ce2-06d2-4680-af9a-58e00ea3ac03","Type":"ContainerDied","Data":"03bf028e254c7000fa4ee9c489d540b0cd885604f906590e493a3d79dca0c575"} Oct 30 19:16:05.914136 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.913970 5959 scope.go:122] "RemoveContainer" containerID="b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b" Oct 30 19:16:05.915749 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.915643 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-podinfo" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:16:05.917016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.916942 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-token" pod "6bd33ce2-06d2-4680-af9a-58e00ea3ac03" (UID: "6bd33ce2-06d2-4680-af9a-58e00ea3ac03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:05.917191 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.917022 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.917307 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.917292 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.917439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.917408 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.917571 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.917440 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lppd\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-kube-api-access-4lppd\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.917571 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.917457 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.917571 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.917471 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.917571 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.917497 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.917571 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.917511 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.928369 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.928339 5959 scope.go:122] "RemoveContainer" containerID="b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b" Oct 30 19:16:05.928849 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:05.928826 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b\": container with ID starting with b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b not found: ID does not exist" containerID="b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b" Oct 30 19:16:05.928982 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:05.928858 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b"} err="failed to get container status \"b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b\": rpc error: code = NotFound desc = could not find container \"b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b\": container with ID starting with b29b44713f8442b6c31aabcfece63008f1a2fcc3aa0850729df29833705f777b not found: ID does not exist" Oct 30 19:16:06.018973 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:06.018936 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd33ce2-06d2-4680-af9a-58e00ea3ac03-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:06.223812 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:06.223775 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg"] Oct 30 19:16:06.228124 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:06.228095 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-qwqmg"] Oct 30 19:16:07.800395 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:07.800360 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" path="/var/lib/kubelet/pods/6bd33ce2-06d2-4680-af9a-58e00ea3ac03/volumes" Oct 30 19:16:07.893883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:07.893811 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm"] Oct 30 19:16:07.898641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:07.898597 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm"] Oct 30 19:16:07.915570 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:07.915537 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml"] Oct 30 19:16:10.003362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:10.003323 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" podUID="23bd45ad-4844-4771-ab04-be34f44f15f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:10.003863 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:10.003534 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" podUID="cff7c99c-0d72-4542-a2ef-28910e83d0bf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:11.154743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:11.154704 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc"] Oct 30 19:16:13.355983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:13.355939 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp"] Oct 30 19:16:13.978582 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:13.978545 5959 generic.go:356] "Generic (PLEG): container finished" podID="f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" containerID="33eda19fdb078aec365aa12b580b264237cafd8922e2c9bc665b257923a73d41" exitCode=0 Oct 30 19:16:13.978787 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:13.978657 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" event={"ID":"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07","Type":"ContainerDied","Data":"33eda19fdb078aec365aa12b580b264237cafd8922e2c9bc665b257923a73d41"} Oct 30 19:16:13.980669 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:13.980641 5959 generic.go:356] "Generic (PLEG): container finished" podID="74f87578-e8f2-4a8a-9e6b-0525fd6c45db" containerID="fd9b34ac8a7794d89bd360cb45d41c0463e7322ba711e58f49f7fd2402e8291a" exitCode=0 Oct 30 19:16:13.981658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:13.980879 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" event={"ID":"74f87578-e8f2-4a8a-9e6b-0525fd6c45db","Type":"ContainerDied","Data":"fd9b34ac8a7794d89bd360cb45d41c0463e7322ba711e58f49f7fd2402e8291a"} Oct 30 19:16:14.102041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.102012 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:16:14.157970 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.157941 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:16:14.293305 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.293274 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-kube-api-access-ggrcx\" (UniqueName: \"kubernetes.io/projected/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-kube-api-access-ggrcx\") pod \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\" (UID: \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\") " Oct 30 19:16:14.293678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.293442 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-plugins-config-volume\") pod \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\" (UID: \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\") " Oct 30 19:16:14.293678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.293559 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-kube-api-access-znxlg\" (UniqueName: \"kubernetes.io/projected/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-kube-api-access-znxlg\") pod \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\" (UID: \"74f87578-e8f2-4a8a-9e6b-0525fd6c45db\") " Oct 30 19:16:14.293678 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.293682 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-plugins-config-volume\") pod \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\" (UID: \"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07\") " Oct 30 19:16:14.294160 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.293973 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-plugins-config-volume" pod "74f87578-e8f2-4a8a-9e6b-0525fd6c45db" (UID: "74f87578-e8f2-4a8a-9e6b-0525fd6c45db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:14.294283 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.294247 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-plugins-config-volume" pod "f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" (UID: "f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:14.294667 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.294645 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:14.294738 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.294674 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:14.296513 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.296492 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-kube-api-access-znxlg" pod "74f87578-e8f2-4a8a-9e6b-0525fd6c45db" (UID: "74f87578-e8f2-4a8a-9e6b-0525fd6c45db"). InnerVolumeSpecName "kube-api-access-znxlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:14.296623 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.296582 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-kube-api-access-ggrcx" pod "f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" (UID: "f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07"). InnerVolumeSpecName "kube-api-access-ggrcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:14.396386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.396248 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxlg\" (UniqueName: \"kubernetes.io/projected/74f87578-e8f2-4a8a-9e6b-0525fd6c45db-kube-api-access-znxlg\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:14.396386 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.396281 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrcx\" (UniqueName: \"kubernetes.io/projected/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07-kube-api-access-ggrcx\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:14.447350 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.447323 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 19:16:14.519333 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.519297 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 19:16:14.599412 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.599380 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff7c99c-0d72-4542-a2ef-28910e83d0bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff7c99c-0d72-4542-a2ef-28910e83d0bf-plugins-config-volume\") pod \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\" (UID: \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\") " Oct 30 19:16:14.599593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.599518 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff7c99c-0d72-4542-a2ef-28910e83d0bf-kube-api-access-md9c9\" (UniqueName: \"kubernetes.io/projected/cff7c99c-0d72-4542-a2ef-28910e83d0bf-kube-api-access-md9c9\") pod \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\" (UID: \"cff7c99c-0d72-4542-a2ef-28910e83d0bf\") " Oct 30 19:16:14.599873 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.599847 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff7c99c-0d72-4542-a2ef-28910e83d0bf-plugins-config-volume" pod "cff7c99c-0d72-4542-a2ef-28910e83d0bf" (UID: "cff7c99c-0d72-4542-a2ef-28910e83d0bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:14.600268 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.600252 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff7c99c-0d72-4542-a2ef-28910e83d0bf-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:14.601843 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.601822 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff7c99c-0d72-4542-a2ef-28910e83d0bf-kube-api-access-md9c9" pod "cff7c99c-0d72-4542-a2ef-28910e83d0bf" (UID: "cff7c99c-0d72-4542-a2ef-28910e83d0bf"). InnerVolumeSpecName "kube-api-access-md9c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:14.701692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.701649 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23bd45ad-4844-4771-ab04-be34f44f15f2-kube-api-access-w2xmh\" (UniqueName: \"kubernetes.io/projected/23bd45ad-4844-4771-ab04-be34f44f15f2-kube-api-access-w2xmh\") pod \"23bd45ad-4844-4771-ab04-be34f44f15f2\" (UID: \"23bd45ad-4844-4771-ab04-be34f44f15f2\") " Oct 30 19:16:14.701894 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.701775 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23bd45ad-4844-4771-ab04-be34f44f15f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bd45ad-4844-4771-ab04-be34f44f15f2-plugins-config-volume\") pod \"23bd45ad-4844-4771-ab04-be34f44f15f2\" (UID: \"23bd45ad-4844-4771-ab04-be34f44f15f2\") " Oct 30 19:16:14.702177 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.702142 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23bd45ad-4844-4771-ab04-be34f44f15f2-plugins-config-volume" pod "23bd45ad-4844-4771-ab04-be34f44f15f2" (UID: "23bd45ad-4844-4771-ab04-be34f44f15f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:14.702332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.702316 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md9c9\" (UniqueName: \"kubernetes.io/projected/cff7c99c-0d72-4542-a2ef-28910e83d0bf-kube-api-access-md9c9\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:14.702388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.702340 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bd45ad-4844-4771-ab04-be34f44f15f2-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:14.703991 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.703971 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bd45ad-4844-4771-ab04-be34f44f15f2-kube-api-access-w2xmh" pod "23bd45ad-4844-4771-ab04-be34f44f15f2" (UID: "23bd45ad-4844-4771-ab04-be34f44f15f2"). InnerVolumeSpecName "kube-api-access-w2xmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:14.804414 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.804383 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xmh\" (UniqueName: \"kubernetes.io/projected/23bd45ad-4844-4771-ab04-be34f44f15f2-kube-api-access-w2xmh\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:14.989210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.989119 5959 generic.go:356] "Generic (PLEG): container finished" podID="cff7c99c-0d72-4542-a2ef-28910e83d0bf" containerID="d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead" exitCode=0 Oct 30 19:16:14.989210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.989169 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" Oct 30 19:16:14.989440 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.989203 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" event={"ID":"cff7c99c-0d72-4542-a2ef-28910e83d0bf","Type":"ContainerDied","Data":"d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead"} Oct 30 19:16:14.989440 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.989254 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp" event={"ID":"cff7c99c-0d72-4542-a2ef-28910e83d0bf","Type":"ContainerDied","Data":"d6446c47bab72b0fc6de393f8c0d634768fea3dd06c09855d4308a276ab2bf0a"} Oct 30 19:16:14.989440 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.989282 5959 scope.go:122] "RemoveContainer" containerID="d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead" Oct 30 19:16:14.991035 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.991000 5959 generic.go:356] "Generic (PLEG): container finished" podID="23bd45ad-4844-4771-ab04-be34f44f15f2" containerID="94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0" exitCode=0 Oct 30 19:16:14.991179 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.991032 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" event={"ID":"23bd45ad-4844-4771-ab04-be34f44f15f2","Type":"ContainerDied","Data":"94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0"} Oct 30 19:16:14.991179 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.991052 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" Oct 30 19:16:14.991179 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.991082 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7" event={"ID":"23bd45ad-4844-4771-ab04-be34f44f15f2","Type":"ContainerDied","Data":"3ac6ed55df7e9172fcc860bbdb8c60c7d50c60f343c7762d23e1ef5b092b53bc"} Oct 30 19:16:14.993028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.993002 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" Oct 30 19:16:14.993028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.993013 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj" event={"ID":"f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07","Type":"ContainerDied","Data":"2f650b0dd91f70885371dfedbf422f4fbefd0afa6bfc4144a24745b43031c5d3"} Oct 30 19:16:14.995188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.995152 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" event={"ID":"74f87578-e8f2-4a8a-9e6b-0525fd6c45db","Type":"ContainerDied","Data":"6fdce402dcedbc6d5e09520174edaa620301b6e777ab0615bf985218ab5ba389"} Oct 30 19:16:14.995188 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:14.995205 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n" Oct 30 19:16:15.003046 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.003020 5959 scope.go:122] "RemoveContainer" containerID="d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead" Oct 30 19:16:15.003432 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:15.003402 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead\": container with ID starting with d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead not found: ID does not exist" containerID="d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead" Oct 30 19:16:15.003560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.003441 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead"} err="failed to get container status \"d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead\": rpc error: code = NotFound desc = could not find container \"d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead\": container with ID starting with d8fbc932e8c730f0b8ca96c0e0f530c034a3539ba834a3e8a9032dce534f0ead not found: ID does not exist" Oct 30 19:16:15.003560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.003537 5959 scope.go:122] "RemoveContainer" containerID="94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0" Oct 30 19:16:15.014763 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.014738 5959 scope.go:122] "RemoveContainer" containerID="94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0" Oct 30 19:16:15.015533 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:15.015498 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0\": container with ID starting with 94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0 not found: ID does not exist" containerID="94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0" Oct 30 19:16:15.015671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.015554 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0"} err="failed to get container status \"94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0\": rpc error: code = NotFound desc = could not find container \"94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0\": container with ID starting with 94df254779bb4cd360ef6ab0efff68d6c402f3b607df33188d9eadeed7a634a0 not found: ID does not exist" Oct 30 19:16:15.015671 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.015582 5959 scope.go:122] "RemoveContainer" containerID="33eda19fdb078aec365aa12b580b264237cafd8922e2c9bc665b257923a73d41" Oct 30 19:16:15.019380 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.019341 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp"] Oct 30 19:16:15.024245 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.024223 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9de872-7f6d476d96-p52rp"] Oct 30 19:16:15.029223 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.029206 5959 scope.go:122] "RemoveContainer" containerID="fd9b34ac8a7794d89bd360cb45d41c0463e7322ba711e58f49f7fd2402e8291a" Oct 30 19:16:15.034173 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.034145 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7"] Oct 30 19:16:15.040950 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.040790 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9de872-667c8d977f-tslk7"] Oct 30 19:16:15.051302 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.051271 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n"] Oct 30 19:16:15.055205 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.055179 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4a477d-6d8d59f968-skf9n"] Oct 30 19:16:15.067824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.067786 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj"] Oct 30 19:16:15.070872 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.070845 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a477d-66d7dcf5d7-l6fzj"] Oct 30 19:16:15.807941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.807905 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23bd45ad-4844-4771-ab04-be34f44f15f2" path="/var/lib/kubelet/pods/23bd45ad-4844-4771-ab04-be34f44f15f2/volumes" Oct 30 19:16:15.808453 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.808378 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f87578-e8f2-4a8a-9e6b-0525fd6c45db" path="/var/lib/kubelet/pods/74f87578-e8f2-4a8a-9e6b-0525fd6c45db/volumes" Oct 30 19:16:15.808783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.808767 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff7c99c-0d72-4542-a2ef-28910e83d0bf" path="/var/lib/kubelet/pods/cff7c99c-0d72-4542-a2ef-28910e83d0bf/volumes" Oct 30 19:16:15.809152 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:15.809138 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" path="/var/lib/kubelet/pods/f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07/volumes" Oct 30 19:16:16.155545 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:16.155495 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" podUID="7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" containerName="payload-processing" containerID="cri-o://94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479" gracePeriod=25 Oct 30 19:16:17.534470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:17.534433 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8"] Oct 30 19:16:17.535371 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:17.535335 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" containerID="cri-o://3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3" gracePeriod=30 Oct 30 19:16:17.543815 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:17.543773 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc"] Oct 30 19:16:17.545154 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:17.544266 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" containerID="cri-o://417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2" gracePeriod=30 Oct 30 19:16:18.356446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:18.356383 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" podUID="7676fec7-190b-4c60-8cbd-aded416ed747" containerName="payload-pre-processing" containerID="cri-o://76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606" gracePeriod=25 Oct 30 19:16:19.461381 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.461341 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb"] Oct 30 19:16:19.461934 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.461572 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" podUID="122408a8-f58c-46bf-b236-5beaf8afd104" containerName="maas-api" containerID="cri-o://1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0" gracePeriod=30 Oct 30 19:16:19.898538 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.898506 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:16:19.908781 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.908748 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-metrics-certs\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-metrics-certs\") pod \"122408a8-f58c-46bf-b236-5beaf8afd104\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " Oct 30 19:16:19.908960 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.908869 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122408a8-f58c-46bf-b236-5beaf8afd104-kube-api-access-fl8mc\" (UniqueName: \"kubernetes.io/projected/122408a8-f58c-46bf-b236-5beaf8afd104-kube-api-access-fl8mc\") pod \"122408a8-f58c-46bf-b236-5beaf8afd104\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " Oct 30 19:16:19.908960 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.908894 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122408a8-f58c-46bf-b236-5beaf8afd104-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/122408a8-f58c-46bf-b236-5beaf8afd104-openshift-service-ca\") pod \"122408a8-f58c-46bf-b236-5beaf8afd104\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " Oct 30 19:16:19.909119 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.909101 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-maas-api-tls\") pod \"122408a8-f58c-46bf-b236-5beaf8afd104\" (UID: \"122408a8-f58c-46bf-b236-5beaf8afd104\") " Oct 30 19:16:19.909349 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.909311 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122408a8-f58c-46bf-b236-5beaf8afd104-openshift-service-ca" pod "122408a8-f58c-46bf-b236-5beaf8afd104" (UID: "122408a8-f58c-46bf-b236-5beaf8afd104"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:19.909861 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.909834 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/122408a8-f58c-46bf-b236-5beaf8afd104-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:19.911301 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.911272 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-metrics-certs" pod "122408a8-f58c-46bf-b236-5beaf8afd104" (UID: "122408a8-f58c-46bf-b236-5beaf8afd104"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:19.911388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.911333 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-maas-api-tls" pod "122408a8-f58c-46bf-b236-5beaf8afd104" (UID: "122408a8-f58c-46bf-b236-5beaf8afd104"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:19.911388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:19.911337 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122408a8-f58c-46bf-b236-5beaf8afd104-kube-api-access-fl8mc" pod "122408a8-f58c-46bf-b236-5beaf8afd104" (UID: "122408a8-f58c-46bf-b236-5beaf8afd104"). InnerVolumeSpecName "kube-api-access-fl8mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:20.016106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.016051 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.016106 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.016097 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl8mc\" (UniqueName: \"kubernetes.io/projected/122408a8-f58c-46bf-b236-5beaf8afd104-kube-api-access-fl8mc\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.016808 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.016115 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/122408a8-f58c-46bf-b236-5beaf8afd104-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.050278 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.050240 5959 generic.go:356] "Generic (PLEG): container finished" podID="122408a8-f58c-46bf-b236-5beaf8afd104" containerID="1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0" exitCode=0 Oct 30 19:16:20.050456 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.050340 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" Oct 30 19:16:20.050456 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.050377 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" event={"ID":"122408a8-f58c-46bf-b236-5beaf8afd104","Type":"ContainerDied","Data":"1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0"} Oct 30 19:16:20.050456 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.050432 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" event={"ID":"122408a8-f58c-46bf-b236-5beaf8afd104","Type":"ContainerDied","Data":"dbc0d9a32eff92c42066aa68b1ac14e01222293cdbd2bdb20006ee785dfcb095"} Oct 30 19:16:20.050581 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.050460 5959 scope.go:122] "RemoveContainer" containerID="1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0" Oct 30 19:16:20.065927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.065901 5959 scope.go:122] "RemoveContainer" containerID="1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0" Oct 30 19:16:20.067048 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:20.066959 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0\": container with ID starting with 1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0 not found: ID does not exist" containerID="1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0" Oct 30 19:16:20.067048 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.067005 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0"} err="failed to get container status \"1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0\": rpc error: code = NotFound desc = could not find container \"1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0\": container with ID starting with 1e3ae5c79cdf5e86a218536096069babebcc8cb06c90d675e9614197e9ae5da0 not found: ID does not exist" Oct 30 19:16:20.093919 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.093888 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb"] Oct 30 19:16:20.100924 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:20.100888 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb"] Oct 30 19:16:21.798057 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:21.798022 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122408a8-f58c-46bf-b236-5beaf8afd104" path="/var/lib/kubelet/pods/122408a8-f58c-46bf-b236-5beaf8afd104/volumes" Oct 30 19:16:21.860853 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:21.860816 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:16:21.861023 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:21.860874 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:16:22.041448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.041410 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" podUID="7676fec7-190b-4c60-8cbd-aded416ed747" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:22.167583 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.167545 5959 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:16:22.167767 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.167642 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:16:22.775704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.775653 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2d1c71-76bcc989b5-nlnjb" podUID="122408a8-f58c-46bf-b236-5beaf8afd104" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.107:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:16:22.857387 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.857357 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 19:16:22.888173 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.888150 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 19:16:22.925020 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.924982 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" containerID="cri-o://7620fecbc327c416e6d96a2e32d80659c3fdb807dc39e88187e9f0460ab7274f" gracePeriod=45 Oct 30 19:16:22.926902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.926854 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" containerID="cri-o://cdca4ce44010bff7282bf11cf2720f8cc09a89df69f52261dc9733cea8bf8c8e" gracePeriod=45 Oct 30 19:16:22.946503 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.946459 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" containerID="cri-o://ef911e14d3567f948ea24ece53e438168b7086ad3894dd257deb32a6e4bda589" gracePeriod=45 Oct 30 19:16:22.977137 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.977093 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 19:16:22.981158 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.981122 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-kube-api-access-5zhh2\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-kube-api-access-5zhh2\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.981383 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.981210 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-data\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.981731 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.981703 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-token\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-token\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.981836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.981805 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-certs\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.981927 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.981905 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-podinfo\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.981990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.981957 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-socket\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.982050 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.981991 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-envoy\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.982050 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982026 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-certs\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.982157 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982061 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-podinfo\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.982157 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982111 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-data\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.982157 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982150 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-envoy\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.982592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982202 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istiod-ca-cert\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.982592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982252 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-credential-socket\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.982592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982302 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-kube-api-access-dz6bl\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-kube-api-access-dz6bl\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.982592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982364 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-socket\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.982592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982410 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-credential-socket\") pod \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\" (UID: \"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873\") " Oct 30 19:16:22.982592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982487 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-token\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-token\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.982592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.982529 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d5d58c-3291-49e5-9e98-e2989a67316a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d5d58c-3291-49e5-9e98-e2989a67316a-istiod-ca-cert\") pod \"45d5d58c-3291-49e5-9e98-e2989a67316a\" (UID: \"45d5d58c-3291-49e5-9e98-e2989a67316a\") " Oct 30 19:16:22.983911 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.983825 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-data" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.984082 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.983972 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istiod-ca-cert" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:22.984082 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.984060 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-socket" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.984226 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.984169 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-certs" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.984887 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.984376 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-credential-socket" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.984887 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.984757 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-socket" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.984887 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.984854 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-credential-socket" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.985100 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.984999 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-certs" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.985239 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.985206 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-data" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.985476 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.985446 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d5d58c-3291-49e5-9e98-e2989a67316a-istiod-ca-cert" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:22.988378 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.988336 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-kube-api-access-5zhh2" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "kube-api-access-5zhh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:22.988646 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.988566 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-token" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:22.991347 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.991309 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-envoy" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.992424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.992344 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-token" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:22.992424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.992364 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-kube-api-access-dz6bl" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "kube-api-access-dz6bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:22.992424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.992367 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-envoy" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:22.992424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.992350 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-podinfo" pod "45d5d58c-3291-49e5-9e98-e2989a67316a" (UID: "45d5d58c-3291-49e5-9e98-e2989a67316a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:16:22.993774 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:22.993752 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-podinfo" pod "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" (UID: "dfb53f1d-da29-4ce3-bd69-19aa0c1dd873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:16:23.081853 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.081752 5959 generic.go:356] "Generic (PLEG): container finished" podID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerID="cdca4ce44010bff7282bf11cf2720f8cc09a89df69f52261dc9733cea8bf8c8e" exitCode=0 Oct 30 19:16:23.082035 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.081851 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" event={"ID":"cd0d64dc-513a-4110-9364-29c7960c74a3","Type":"ContainerDied","Data":"cdca4ce44010bff7282bf11cf2720f8cc09a89df69f52261dc9733cea8bf8c8e"} Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.083755 5959 generic.go:356] "Generic (PLEG): container finished" podID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerID="3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3" exitCode=0 Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.083917 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" event={"ID":"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873","Type":"ContainerDied","Data":"3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3"} Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.083965 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" event={"ID":"dfb53f1d-da29-4ce3-bd69-19aa0c1dd873","Type":"ContainerDied","Data":"eaa789cbf231c5ab968e5a3cf6059a254966ca0b13068236762cb394d232f9c6"} Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.083994 5959 scope.go:122] "RemoveContainer" containerID="3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084117 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084779 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084807 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084822 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084836 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084853 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084868 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084882 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084894 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084907 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084922 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz6bl\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-kube-api-access-dz6bl\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084940 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084954 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084968 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084984 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d5d58c-3291-49e5-9e98-e2989a67316a-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.084998 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhh2\" (UniqueName: \"kubernetes.io/projected/45d5d58c-3291-49e5-9e98-e2989a67316a-kube-api-access-5zhh2\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.085012 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.085025 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.085326 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.085039 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d5d58c-3291-49e5-9e98-e2989a67316a-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.090751 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.090461 5959 generic.go:356] "Generic (PLEG): container finished" podID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerID="7620fecbc327c416e6d96a2e32d80659c3fdb807dc39e88187e9f0460ab7274f" exitCode=0 Oct 30 19:16:23.090751 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.090579 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" event={"ID":"28ae7e50-6451-4b7e-8691-999aa3234d94","Type":"ContainerDied","Data":"7620fecbc327c416e6d96a2e32d80659c3fdb807dc39e88187e9f0460ab7274f"} Oct 30 19:16:23.094461 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.093924 5959 generic.go:356] "Generic (PLEG): container finished" podID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerID="ef911e14d3567f948ea24ece53e438168b7086ad3894dd257deb32a6e4bda589" exitCode=0 Oct 30 19:16:23.094461 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.094069 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" event={"ID":"36cc2519-050e-4689-975a-b33a9dd76d5e","Type":"ContainerDied","Data":"ef911e14d3567f948ea24ece53e438168b7086ad3894dd257deb32a6e4bda589"} Oct 30 19:16:23.100661 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.100579 5959 generic.go:356] "Generic (PLEG): container finished" podID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerID="417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2" exitCode=0 Oct 30 19:16:23.100801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.100694 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" Oct 30 19:16:23.100801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.100723 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" event={"ID":"45d5d58c-3291-49e5-9e98-e2989a67316a","Type":"ContainerDied","Data":"417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2"} Oct 30 19:16:23.100801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.100771 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc" event={"ID":"45d5d58c-3291-49e5-9e98-e2989a67316a","Type":"ContainerDied","Data":"33f5d2fcd29ea8b0c2d245b29fb4d3ff5cc50f00f214e121a96c3a229180b8ea"} Oct 30 19:16:23.107945 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.107834 5959 scope.go:122] "RemoveContainer" containerID="3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3" Oct 30 19:16:23.110628 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:23.110577 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3\": container with ID starting with 3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3 not found: ID does not exist" containerID="3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3" Oct 30 19:16:23.110897 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.110777 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3"} err="failed to get container status \"3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3\": rpc error: code = NotFound desc = could not find container \"3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3\": container with ID starting with 3655233a30a02924689258da8d31d18d9a06d2e132ca3364a7051abfcbc83db3 not found: ID does not exist" Oct 30 19:16:23.110897 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.110806 5959 scope.go:122] "RemoveContainer" containerID="417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2" Oct 30 19:16:23.122561 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.122502 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8"] Oct 30 19:16:23.128915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.128890 5959 scope.go:122] "RemoveContainer" containerID="417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2" Oct 30 19:16:23.129358 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:23.129329 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2\": container with ID starting with 417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2 not found: ID does not exist" containerID="417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2" Oct 30 19:16:23.129432 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.129370 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2"} err="failed to get container status \"417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2\": rpc error: code = NotFound desc = could not find container \"417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2\": container with ID starting with 417eb879f2070dbc2b2869febf9cfb5263f469936afa04389b649cdc434eccd2 not found: ID does not exist" Oct 30 19:16:23.129840 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.129809 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-g95k8"] Oct 30 19:16:23.140621 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.140565 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc"] Oct 30 19:16:23.149345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.149273 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-67wlc"] Oct 30 19:16:23.286824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.272728 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:16:23.309549 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.309520 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:16:23.317189 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.317157 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:16:23.397101 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397069 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-model-cache\") pod \"36cc2519-050e-4689-975a-b33a9dd76d5e\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " Oct 30 19:16:23.397325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397107 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ae7e50-6451-4b7e-8691-999aa3234d94-kube-api-access-ldg2b\" (UniqueName: \"kubernetes.io/projected/28ae7e50-6451-4b7e-8691-999aa3234d94-kube-api-access-ldg2b\") pod \"28ae7e50-6451-4b7e-8691-999aa3234d94\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " Oct 30 19:16:23.397325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397140 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-home\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-home\") pod \"cd0d64dc-513a-4110-9364-29c7960c74a3\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " Oct 30 19:16:23.397325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397163 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-tmp-dir\") pod \"cd0d64dc-513a-4110-9364-29c7960c74a3\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " Oct 30 19:16:23.397325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397217 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-tmp-dir\") pod \"28ae7e50-6451-4b7e-8691-999aa3234d94\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " Oct 30 19:16:23.397325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397239 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0d64dc-513a-4110-9364-29c7960c74a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0d64dc-513a-4110-9364-29c7960c74a3-tls-certs\") pod \"cd0d64dc-513a-4110-9364-29c7960c74a3\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " Oct 30 19:16:23.397325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397289 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-home\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-home\") pod \"36cc2519-050e-4689-975a-b33a9dd76d5e\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397336 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-model-cache\") pod \"28ae7e50-6451-4b7e-8691-999aa3234d94\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397365 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-model-cache\") pod \"cd0d64dc-513a-4110-9364-29c7960c74a3\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397437 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-home" pod "cd0d64dc-513a-4110-9364-29c7960c74a3" (UID: "cd0d64dc-513a-4110-9364-29c7960c74a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397442 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-model-cache" pod "36cc2519-050e-4689-975a-b33a9dd76d5e" (UID: "36cc2519-050e-4689-975a-b33a9dd76d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397497 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-tmp-dir" pod "cd0d64dc-513a-4110-9364-29c7960c74a3" (UID: "cd0d64dc-513a-4110-9364-29c7960c74a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397448 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0d64dc-513a-4110-9364-29c7960c74a3-kube-api-access-kkvrb\" (UniqueName: \"kubernetes.io/projected/cd0d64dc-513a-4110-9364-29c7960c74a3-kube-api-access-kkvrb\") pod \"cd0d64dc-513a-4110-9364-29c7960c74a3\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397518 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-tmp-dir" pod "28ae7e50-6451-4b7e-8691-999aa3234d94" (UID: "28ae7e50-6451-4b7e-8691-999aa3234d94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397628 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-dshm\") pod \"28ae7e50-6451-4b7e-8691-999aa3234d94\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397661 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-home" pod "36cc2519-050e-4689-975a-b33a9dd76d5e" (UID: "36cc2519-050e-4689-975a-b33a9dd76d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.397692 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397675 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cc2519-050e-4689-975a-b33a9dd76d5e-kube-api-access-6hg86\" (UniqueName: \"kubernetes.io/projected/36cc2519-050e-4689-975a-b33a9dd76d5e-kube-api-access-6hg86\") pod \"36cc2519-050e-4689-975a-b33a9dd76d5e\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " Oct 30 19:16:23.398256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397708 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36cc2519-050e-4689-975a-b33a9dd76d5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/36cc2519-050e-4689-975a-b33a9dd76d5e-tls-certs\") pod \"36cc2519-050e-4689-975a-b33a9dd76d5e\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " Oct 30 19:16:23.398256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397792 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-model-cache" pod "cd0d64dc-513a-4110-9364-29c7960c74a3" (UID: "cd0d64dc-513a-4110-9364-29c7960c74a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.398256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397831 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-model-cache" pod "28ae7e50-6451-4b7e-8691-999aa3234d94" (UID: "28ae7e50-6451-4b7e-8691-999aa3234d94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.398256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397885 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-dshm\") pod \"36cc2519-050e-4689-975a-b33a9dd76d5e\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " Oct 30 19:16:23.398256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397918 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ae7e50-6451-4b7e-8691-999aa3234d94-tls-certs\" (UniqueName: \"kubernetes.io/secret/28ae7e50-6451-4b7e-8691-999aa3234d94-tls-certs\") pod \"28ae7e50-6451-4b7e-8691-999aa3234d94\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " Oct 30 19:16:23.398256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.397996 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-dshm\") pod \"cd0d64dc-513a-4110-9364-29c7960c74a3\" (UID: \"cd0d64dc-513a-4110-9364-29c7960c74a3\") " Oct 30 19:16:23.398256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.398016 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-home\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-home\") pod \"28ae7e50-6451-4b7e-8691-999aa3234d94\" (UID: \"28ae7e50-6451-4b7e-8691-999aa3234d94\") " Oct 30 19:16:23.398256 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.398056 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-tmp-dir\") pod \"36cc2519-050e-4689-975a-b33a9dd76d5e\" (UID: \"36cc2519-050e-4689-975a-b33a9dd76d5e\") " Oct 30 19:16:23.399194 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399167 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.399194 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399195 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.399377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399210 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.399377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399224 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.399377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399236 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.399377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399248 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.399377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399262 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.399777 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399488 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-tmp-dir" pod "36cc2519-050e-4689-975a-b33a9dd76d5e" (UID: "36cc2519-050e-4689-975a-b33a9dd76d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.399777 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.399536 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-home" pod "28ae7e50-6451-4b7e-8691-999aa3234d94" (UID: "28ae7e50-6451-4b7e-8691-999aa3234d94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.402157 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.402099 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ae7e50-6451-4b7e-8691-999aa3234d94-kube-api-access-ldg2b" pod "28ae7e50-6451-4b7e-8691-999aa3234d94" (UID: "28ae7e50-6451-4b7e-8691-999aa3234d94"). InnerVolumeSpecName "kube-api-access-ldg2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:23.402157 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.402134 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0d64dc-513a-4110-9364-29c7960c74a3-kube-api-access-kkvrb" pod "cd0d64dc-513a-4110-9364-29c7960c74a3" (UID: "cd0d64dc-513a-4110-9364-29c7960c74a3"). InnerVolumeSpecName "kube-api-access-kkvrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:23.402804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.402723 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36cc2519-050e-4689-975a-b33a9dd76d5e-tls-certs" pod "36cc2519-050e-4689-975a-b33a9dd76d5e" (UID: "36cc2519-050e-4689-975a-b33a9dd76d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:23.402804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.402765 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-dshm" pod "28ae7e50-6451-4b7e-8691-999aa3234d94" (UID: "28ae7e50-6451-4b7e-8691-999aa3234d94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.402804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.402763 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0d64dc-513a-4110-9364-29c7960c74a3-tls-certs" pod "cd0d64dc-513a-4110-9364-29c7960c74a3" (UID: "cd0d64dc-513a-4110-9364-29c7960c74a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:23.402804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.402789 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ae7e50-6451-4b7e-8691-999aa3234d94-tls-certs" pod "28ae7e50-6451-4b7e-8691-999aa3234d94" (UID: "28ae7e50-6451-4b7e-8691-999aa3234d94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:23.403989 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.403964 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cc2519-050e-4689-975a-b33a9dd76d5e-kube-api-access-6hg86" pod "36cc2519-050e-4689-975a-b33a9dd76d5e" (UID: "36cc2519-050e-4689-975a-b33a9dd76d5e"). InnerVolumeSpecName "kube-api-access-6hg86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:23.404070 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.404030 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-dshm" pod "36cc2519-050e-4689-975a-b33a9dd76d5e" (UID: "36cc2519-050e-4689-975a-b33a9dd76d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.404465 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.404438 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-dshm" pod "cd0d64dc-513a-4110-9364-29c7960c74a3" (UID: "cd0d64dc-513a-4110-9364-29c7960c74a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:23.500537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500503 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500534 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ae7e50-6451-4b7e-8691-999aa3234d94-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500537 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500544 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0d64dc-513a-4110-9364-29c7960c74a3-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500552 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500562 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cc2519-050e-4689-975a-b33a9dd76d5e-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500571 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldg2b\" (UniqueName: \"kubernetes.io/projected/28ae7e50-6451-4b7e-8691-999aa3234d94-kube-api-access-ldg2b\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500579 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0d64dc-513a-4110-9364-29c7960c74a3-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500588 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvrb\" (UniqueName: \"kubernetes.io/projected/cd0d64dc-513a-4110-9364-29c7960c74a3-kube-api-access-kkvrb\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500596 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ae7e50-6451-4b7e-8691-999aa3234d94-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500642 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hg86\" (UniqueName: \"kubernetes.io/projected/36cc2519-050e-4689-975a-b33a9dd76d5e-kube-api-access-6hg86\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.500845 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.500654 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cc2519-050e-4689-975a-b33a9dd76d5e-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:23.800088 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.800005 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" path="/var/lib/kubelet/pods/45d5d58c-3291-49e5-9e98-e2989a67316a/volumes" Oct 30 19:16:23.800701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:23.800681 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" path="/var/lib/kubelet/pods/dfb53f1d-da29-4ce3-bd69-19aa0c1dd873/volumes" Oct 30 19:16:24.109469 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.109375 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" Oct 30 19:16:24.109469 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.109409 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm" event={"ID":"28ae7e50-6451-4b7e-8691-999aa3234d94","Type":"ContainerDied","Data":"02f7ab40f7436f19fd071368fff39df8e0accd795cfc76a40bfd641b1207e42c"} Oct 30 19:16:24.110004 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.109471 5959 scope.go:122] "RemoveContainer" containerID="7620fecbc327c416e6d96a2e32d80659c3fdb807dc39e88187e9f0460ab7274f" Oct 30 19:16:24.112410 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.111493 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" event={"ID":"36cc2519-050e-4689-975a-b33a9dd76d5e","Type":"ContainerDied","Data":"b50035723a7a3a3b997bdc396c8ec9ca70a49db60dc7bed02f0b6e786c65b0cd"} Oct 30 19:16:24.112410 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.111592 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" Oct 30 19:16:24.114855 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.114834 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" Oct 30 19:16:24.114855 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.114848 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm" event={"ID":"cd0d64dc-513a-4110-9364-29c7960c74a3","Type":"ContainerDied","Data":"88c25b21cef4223dabfc83c5aa2492b15d2c159b70a0c47361c9fc4264db926d"} Oct 30 19:16:24.121320 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.121300 5959 scope.go:122] "RemoveContainer" containerID="ef911e14d3567f948ea24ece53e438168b7086ad3894dd257deb32a6e4bda589" Oct 30 19:16:24.130792 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.130761 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm"] Oct 30 19:16:24.132941 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.132922 5959 scope.go:122] "RemoveContainer" containerID="cdca4ce44010bff7282bf11cf2720f8cc09a89df69f52261dc9733cea8bf8c8e" Oct 30 19:16:24.135249 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.135227 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/distinct-w4-0458df-kserve-c586879bb-r6wlm"] Oct 30 19:16:24.144495 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.144458 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml"] Oct 30 19:16:24.147706 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.147679 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml"] Oct 30 19:16:24.155949 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.155921 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm"] Oct 30 19:16:24.160174 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.160149 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0458df/premium-w4-0458df-kserve-5d48c994f9-9t6wm"] Oct 30 19:16:24.197969 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.197922 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0458df/simulator-w4-0458df-kserve-6885b74b94-8g6ml" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:16:24.651527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:24.651492 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" podUID="7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:25.799231 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:25.799195 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" path="/var/lib/kubelet/pods/28ae7e50-6451-4b7e-8691-999aa3234d94/volumes" Oct 30 19:16:25.799737 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:25.799720 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" path="/var/lib/kubelet/pods/36cc2519-050e-4689-975a-b33a9dd76d5e/volumes" Oct 30 19:16:25.800165 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:25.800151 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" path="/var/lib/kubelet/pods/cd0d64dc-513a-4110-9364-29c7960c74a3/volumes" Oct 30 19:16:27.797585 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:27.797545 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:16:31.556243 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.556206 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq"] Oct 30 19:16:31.556808 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.556476 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" podUID="d0da26a2-90ab-43e0-9476-bcb1620532b8" containerName="maas-api" containerID="cri-o://0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc" gracePeriod=30 Oct 30 19:16:31.946782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.946742 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:16:31.972703 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.972666 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls\") pod \"d0da26a2-90ab-43e0-9476-bcb1620532b8\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " Oct 30 19:16:31.972909 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.972863 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-metrics-certs\") pod \"d0da26a2-90ab-43e0-9476-bcb1620532b8\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " Oct 30 19:16:31.973089 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.973066 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0da26a2-90ab-43e0-9476-bcb1620532b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da26a2-90ab-43e0-9476-bcb1620532b8-openshift-service-ca\") pod \"d0da26a2-90ab-43e0-9476-bcb1620532b8\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " Oct 30 19:16:31.973236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.973130 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0da26a2-90ab-43e0-9476-bcb1620532b8-kube-api-access-f8q68\" (UniqueName: \"kubernetes.io/projected/d0da26a2-90ab-43e0-9476-bcb1620532b8-kube-api-access-f8q68\") pod \"d0da26a2-90ab-43e0-9476-bcb1620532b8\" (UID: \"d0da26a2-90ab-43e0-9476-bcb1620532b8\") " Oct 30 19:16:31.973684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.973645 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0da26a2-90ab-43e0-9476-bcb1620532b8-openshift-service-ca" pod "d0da26a2-90ab-43e0-9476-bcb1620532b8" (UID: "d0da26a2-90ab-43e0-9476-bcb1620532b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:31.973868 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.973845 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da26a2-90ab-43e0-9476-bcb1620532b8-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:31.976264 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.976229 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0da26a2-90ab-43e0-9476-bcb1620532b8-kube-api-access-f8q68" pod "d0da26a2-90ab-43e0-9476-bcb1620532b8" (UID: "d0da26a2-90ab-43e0-9476-bcb1620532b8"). InnerVolumeSpecName "kube-api-access-f8q68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:31.976406 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.976302 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls" pod "d0da26a2-90ab-43e0-9476-bcb1620532b8" (UID: "d0da26a2-90ab-43e0-9476-bcb1620532b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:31.978407 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:31.978384 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-metrics-certs" pod "d0da26a2-90ab-43e0-9476-bcb1620532b8" (UID: "d0da26a2-90ab-43e0-9476-bcb1620532b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:32.041590 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.041554 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" podUID="7676fec7-190b-4c60-8cbd-aded416ed747" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:32.074961 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.074932 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:32.074961 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.074959 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8q68\" (UniqueName: \"kubernetes.io/projected/d0da26a2-90ab-43e0-9476-bcb1620532b8-kube-api-access-f8q68\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:32.075175 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.074970 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0da26a2-90ab-43e0-9476-bcb1620532b8-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:32.200227 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.200131 5959 generic.go:356] "Generic (PLEG): container finished" podID="d0da26a2-90ab-43e0-9476-bcb1620532b8" containerID="0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc" exitCode=0 Oct 30 19:16:32.200227 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.200181 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" Oct 30 19:16:32.200227 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.200209 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" event={"ID":"d0da26a2-90ab-43e0-9476-bcb1620532b8","Type":"ContainerDied","Data":"0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc"} Oct 30 19:16:32.200467 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.200268 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq" event={"ID":"d0da26a2-90ab-43e0-9476-bcb1620532b8","Type":"ContainerDied","Data":"ea2de3d48ee2b5b676049cf7eb9f5438a39ce4be989bfa6254fadfdfc778610e"} Oct 30 19:16:32.200467 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.200292 5959 scope.go:122] "RemoveContainer" containerID="0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc" Oct 30 19:16:32.211590 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.211568 5959 scope.go:122] "RemoveContainer" containerID="0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc" Oct 30 19:16:32.211967 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:32.211943 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc\": container with ID starting with 0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc not found: ID does not exist" containerID="0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc" Oct 30 19:16:32.212021 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.211978 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc"} err="failed to get container status \"0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc\": rpc error: code = NotFound desc = could not find container \"0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc\": container with ID starting with 0ac2d490ea54d89b77e7a20009164978559087f9d79171e604f6d1b0f96622cc not found: ID does not exist" Oct 30 19:16:32.225111 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.225079 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq"] Oct 30 19:16:32.230496 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:32.230469 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca2cd5-6949b68679-ksddq"] Oct 30 19:16:33.799510 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:33.799471 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0da26a2-90ab-43e0-9476-bcb1620532b8" path="/var/lib/kubelet/pods/d0da26a2-90ab-43e0-9476-bcb1620532b8/volumes" Oct 30 19:16:34.651028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:34.650976 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" podUID="7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:36.450754 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:36.450722 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:16:36.562501 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:36.562461 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-kube-api-access-pvfzd\" (UniqueName: \"kubernetes.io/projected/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-kube-api-access-pvfzd\") pod \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\" (UID: \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\") " Oct 30 19:16:36.562741 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:36.562514 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-plugins-config-volume\") pod \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\" (UID: \"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7\") " Oct 30 19:16:36.563063 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:36.563031 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-plugins-config-volume" pod "7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" (UID: "7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:36.563192 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:36.563177 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:36.565009 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:36.564981 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-kube-api-access-pvfzd" pod "7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" (UID: "7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7"). InnerVolumeSpecName "kube-api-access-pvfzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:36.664313 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:36.664275 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvfzd\" (UniqueName: \"kubernetes.io/projected/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7-kube-api-access-pvfzd\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:37.253066 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.253017 5959 generic.go:356] "Generic (PLEG): container finished" podID="7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" containerID="94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479" exitCode=0 Oct 30 19:16:37.253292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.253123 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" event={"ID":"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7","Type":"ContainerDied","Data":"94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479"} Oct 30 19:16:37.253292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.253175 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" event={"ID":"7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7","Type":"ContainerDied","Data":"962958978fd28341e6cec3a7046004b5498efc3c9470bfd4a9aa78265f1af478"} Oct 30 19:16:37.253292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.253204 5959 scope.go:122] "RemoveContainer" containerID="94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479" Oct 30 19:16:37.253471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.253343 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc" Oct 30 19:16:37.266476 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.266449 5959 scope.go:122] "RemoveContainer" containerID="94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479" Oct 30 19:16:37.266819 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:37.266796 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479\": container with ID starting with 94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479 not found: ID does not exist" containerID="94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479" Oct 30 19:16:37.266906 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.266830 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479"} err="failed to get container status \"94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479\": rpc error: code = NotFound desc = could not find container \"94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479\": container with ID starting with 94212b88e709aa300e92fede04b100745d60915aadc5b65bc7332b05e6db2479 not found: ID does not exist" Oct 30 19:16:37.284776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.284741 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc"] Oct 30 19:16:37.290054 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.290022 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2d1c71-7b84bb8b7b-slhsc"] Oct 30 19:16:37.798550 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:37.798511 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" path="/var/lib/kubelet/pods/7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7/volumes" Oct 30 19:16:38.645152 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:38.645122 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 19:16:38.696206 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:38.696165 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7676fec7-190b-4c60-8cbd-aded416ed747-kube-api-access-hfxw9\" (UniqueName: \"kubernetes.io/projected/7676fec7-190b-4c60-8cbd-aded416ed747-kube-api-access-hfxw9\") pod \"7676fec7-190b-4c60-8cbd-aded416ed747\" (UID: \"7676fec7-190b-4c60-8cbd-aded416ed747\") " Oct 30 19:16:38.696491 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:38.696460 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7676fec7-190b-4c60-8cbd-aded416ed747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7676fec7-190b-4c60-8cbd-aded416ed747-plugins-config-volume\") pod \"7676fec7-190b-4c60-8cbd-aded416ed747\" (UID: \"7676fec7-190b-4c60-8cbd-aded416ed747\") " Oct 30 19:16:38.697923 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:38.697150 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7676fec7-190b-4c60-8cbd-aded416ed747-plugins-config-volume" pod "7676fec7-190b-4c60-8cbd-aded416ed747" (UID: "7676fec7-190b-4c60-8cbd-aded416ed747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:38.698704 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:38.698679 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7676fec7-190b-4c60-8cbd-aded416ed747-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:38.700730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:38.700706 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7676fec7-190b-4c60-8cbd-aded416ed747-kube-api-access-hfxw9" pod "7676fec7-190b-4c60-8cbd-aded416ed747" (UID: "7676fec7-190b-4c60-8cbd-aded416ed747"). InnerVolumeSpecName "kube-api-access-hfxw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:38.807630 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:38.801746 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfxw9\" (UniqueName: \"kubernetes.io/projected/7676fec7-190b-4c60-8cbd-aded416ed747-kube-api-access-hfxw9\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:39.270934 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.270888 5959 generic.go:356] "Generic (PLEG): container finished" podID="7676fec7-190b-4c60-8cbd-aded416ed747" containerID="76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606" exitCode=0 Oct 30 19:16:39.270934 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.270938 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" Oct 30 19:16:39.271321 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.270972 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" event={"ID":"7676fec7-190b-4c60-8cbd-aded416ed747","Type":"ContainerDied","Data":"76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606"} Oct 30 19:16:39.271321 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.271029 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp" event={"ID":"7676fec7-190b-4c60-8cbd-aded416ed747","Type":"ContainerDied","Data":"27e085caaa18c0a2c45e58da6dca7c9e7dad4f86e614d02988693ff455a47a7c"} Oct 30 19:16:39.271321 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.271052 5959 scope.go:122] "RemoveContainer" containerID="76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606" Oct 30 19:16:39.282202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.282182 5959 scope.go:122] "RemoveContainer" containerID="76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606" Oct 30 19:16:39.282504 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:39.282486 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606\": container with ID starting with 76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606 not found: ID does not exist" containerID="76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606" Oct 30 19:16:39.282551 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.282513 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606"} err="failed to get container status \"76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606\": rpc error: code = NotFound desc = could not find container \"76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606\": container with ID starting with 76f16fb64033dd3d13037871c32d5a15804eabf74db24c68ad53a147f3f44606 not found: ID does not exist" Oct 30 19:16:39.298947 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.298916 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp"] Oct 30 19:16:39.302318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.302296 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0458df-59fd997f55-dh6kp"] Oct 30 19:16:39.797182 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.797146 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7676fec7-190b-4c60-8cbd-aded416ed747" path="/var/lib/kubelet/pods/7676fec7-190b-4c60-8cbd-aded416ed747/volumes" Oct 30 19:16:39.924976 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:39.924940 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:16:40.362391 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:40.362351 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6"] Oct 30 19:16:40.379540 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:40.379501 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v"] Oct 30 19:16:44.318786 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:44.318720 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h"] Oct 30 19:16:44.319449 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:44.319020 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" containerID="cri-o://0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360" gracePeriod=30 Oct 30 19:16:44.333093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:44.333053 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq"] Oct 30 19:16:44.333331 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:44.333305 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" containerID="cri-o://6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208" gracePeriod=30 Oct 30 19:16:44.945874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:44.945834 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz"] Oct 30 19:16:44.946129 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:44.946100 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" podUID="0aa10346-b095-4bcb-83c9-b09741af930d" containerName="maas-api" containerID="cri-o://ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715" gracePeriod=30 Oct 30 19:16:45.281122 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.281078 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:16:45.339426 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.339386 5959 generic.go:356] "Generic (PLEG): container finished" podID="0aa10346-b095-4bcb-83c9-b09741af930d" containerID="ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715" exitCode=0 Oct 30 19:16:45.340018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.339455 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" Oct 30 19:16:45.340018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.339579 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" event={"ID":"0aa10346-b095-4bcb-83c9-b09741af930d","Type":"ContainerDied","Data":"ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715"} Oct 30 19:16:45.340018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.339680 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz" event={"ID":"0aa10346-b095-4bcb-83c9-b09741af930d","Type":"ContainerDied","Data":"1e051cb944aa5cbe98754bb8e266a43d2ffcb2f190aee0d3ff08116aad7dbb40"} Oct 30 19:16:45.340018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.339718 5959 scope.go:122] "RemoveContainer" containerID="ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715" Oct 30 19:16:45.352865 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.352833 5959 scope.go:122] "RemoveContainer" containerID="ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715" Oct 30 19:16:45.353266 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:45.353241 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715\": container with ID starting with ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715 not found: ID does not exist" containerID="ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715" Oct 30 19:16:45.353332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.353276 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715"} err="failed to get container status \"ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715\": rpc error: code = NotFound desc = could not find container \"ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715\": container with ID starting with ca384c917045038aaf7160e06dd8d9dd2d56c9cc85999e513ad48b6e5a7eb715 not found: ID does not exist" Oct 30 19:16:45.357815 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.357789 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-metrics-certs\") pod \"0aa10346-b095-4bcb-83c9-b09741af930d\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " Oct 30 19:16:45.357942 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.357886 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa10346-b095-4bcb-83c9-b09741af930d-kube-api-access-vrx5m\" (UniqueName: \"kubernetes.io/projected/0aa10346-b095-4bcb-83c9-b09741af930d-kube-api-access-vrx5m\") pod \"0aa10346-b095-4bcb-83c9-b09741af930d\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " Oct 30 19:16:45.358102 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.358084 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls\") pod \"0aa10346-b095-4bcb-83c9-b09741af930d\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " Oct 30 19:16:45.358177 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.358162 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa10346-b095-4bcb-83c9-b09741af930d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa10346-b095-4bcb-83c9-b09741af930d-openshift-service-ca\") pod \"0aa10346-b095-4bcb-83c9-b09741af930d\" (UID: \"0aa10346-b095-4bcb-83c9-b09741af930d\") " Oct 30 19:16:45.358662 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.358595 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa10346-b095-4bcb-83c9-b09741af930d-openshift-service-ca" pod "0aa10346-b095-4bcb-83c9-b09741af930d" (UID: "0aa10346-b095-4bcb-83c9-b09741af930d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:45.358954 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.358936 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa10346-b095-4bcb-83c9-b09741af930d-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:45.360294 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.360261 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-metrics-certs" pod "0aa10346-b095-4bcb-83c9-b09741af930d" (UID: "0aa10346-b095-4bcb-83c9-b09741af930d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:45.360411 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.360263 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa10346-b095-4bcb-83c9-b09741af930d-kube-api-access-vrx5m" pod "0aa10346-b095-4bcb-83c9-b09741af930d" (UID: "0aa10346-b095-4bcb-83c9-b09741af930d"). InnerVolumeSpecName "kube-api-access-vrx5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:45.360411 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.360323 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls" pod "0aa10346-b095-4bcb-83c9-b09741af930d" (UID: "0aa10346-b095-4bcb-83c9-b09741af930d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:45.363408 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.363378 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" podUID="f2581e0a-b5b3-4df8-8b36-7f9b5232addc" containerName="payload-processing" containerID="cri-o://dcce2ff4b1ba68c4e146a3c3506a3e40ae67262aabfd3c2dbbff95b37adb21c2" gracePeriod=25 Oct 30 19:16:45.380505 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.380464 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" podUID="cb7d1645-e7a7-433a-a39f-8d53f47d01ef" containerName="payload-pre-processing" containerID="cri-o://926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140" gracePeriod=25 Oct 30 19:16:45.460098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.460003 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:45.460098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.460034 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa10346-b095-4bcb-83c9-b09741af930d-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:45.460098 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.460045 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrx5m\" (UniqueName: \"kubernetes.io/projected/0aa10346-b095-4bcb-83c9-b09741af930d-kube-api-access-vrx5m\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:45.664685 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.664640 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz"] Oct 30 19:16:45.678079 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.678036 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef6be7-79f5bb569-hslfz"] Oct 30 19:16:45.798332 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:45.798251 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa10346-b095-4bcb-83c9-b09741af930d" path="/var/lib/kubelet/pods/0aa10346-b095-4bcb-83c9-b09741af930d/volumes" Oct 30 19:16:49.660690 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.660590 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 19:16:49.699156 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.699114 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 19:16:49.716583 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.716540 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-envoy\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.716797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.716690 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-socket\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.716797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.716733 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-token\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-token\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.716797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.716777 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-kube-api-access-vkd69\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-kube-api-access-vkd69\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.716986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.716811 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-socket\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.716986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.716881 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-kube-api-access-h2sxk\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-kube-api-access-h2sxk\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.716986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.716914 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-certs\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.716986 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.716940 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istiod-ca-cert\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.717197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717056 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-certs\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.717197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717093 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istiod-ca-cert\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.717197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717117 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-token\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-token\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.717197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717144 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-podinfo\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.717197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717181 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-data\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.717644 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717213 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-credential-socket\") pod \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\" (UID: \"aea10d0b-617b-4aa0-a73d-5d22a0e7153a\") " Oct 30 19:16:49.717644 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717263 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-data\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.717644 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717298 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-credential-socket\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.717644 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717349 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-podinfo\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.717644 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717360 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-certs" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.717644 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717394 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-envoy\") pod \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\" (UID: \"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f\") " Oct 30 19:16:49.717978 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.717950 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-socket" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.718123 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.718102 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.718189 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.718128 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.718771 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.718675 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-socket" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.718916 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.718796 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-data" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.718994 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.718958 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-credential-socket" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.719261 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.719131 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-data" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.719439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.719415 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-credential-socket" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.720032 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.720006 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-certs" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.720361 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.720335 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istiod-ca-cert" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:49.720691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.720665 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istiod-ca-cert" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:49.725418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.723322 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-podinfo" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:16:49.725418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.723397 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-kube-api-access-vkd69" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "kube-api-access-vkd69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:49.725418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.723864 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-envoy" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.725418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.723875 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-token" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:49.725418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.724159 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-envoy" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:49.725418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.724763 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-podinfo" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:16:49.725418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.725851 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-token" pod "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" (UID: "c9e4ec7d-7b73-42cc-94d3-5eb46d09872f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:49.728782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.728750 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-kube-api-access-h2sxk" pod "aea10d0b-617b-4aa0-a73d-5d22a0e7153a" (UID: "aea10d0b-617b-4aa0-a73d-5d22a0e7153a"). InnerVolumeSpecName "kube-api-access-h2sxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:49.820018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.819986 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820018 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820015 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2sxk\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-kube-api-access-h2sxk\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820026 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820035 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820045 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820054 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820062 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820071 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820079 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820087 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820095 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820103 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820113 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820121 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea10d0b-617b-4aa0-a73d-5d22a0e7153a-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820129 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:49.820253 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:49.820137 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkd69\" (UniqueName: \"kubernetes.io/projected/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f-kube-api-access-vkd69\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:50.378051 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.378008 5959 generic.go:356] "Generic (PLEG): container finished" podID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerID="0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360" exitCode=0 Oct 30 19:16:50.378223 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.378199 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" Oct 30 19:16:50.378858 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.378712 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" event={"ID":"aea10d0b-617b-4aa0-a73d-5d22a0e7153a","Type":"ContainerDied","Data":"0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360"} Oct 30 19:16:50.378858 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.378779 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h" event={"ID":"aea10d0b-617b-4aa0-a73d-5d22a0e7153a","Type":"ContainerDied","Data":"e88310e014b8fec2ce23bd1d5e8615d5a7367c3593de0492440f7acab71c880d"} Oct 30 19:16:50.378858 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.378809 5959 scope.go:122] "RemoveContainer" containerID="0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360" Oct 30 19:16:50.380482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.380455 5959 generic.go:356] "Generic (PLEG): container finished" podID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerID="6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208" exitCode=0 Oct 30 19:16:50.380637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.380525 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" Oct 30 19:16:50.380637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.380567 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" event={"ID":"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f","Type":"ContainerDied","Data":"6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208"} Oct 30 19:16:50.380774 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.380644 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq" event={"ID":"c9e4ec7d-7b73-42cc-94d3-5eb46d09872f","Type":"ContainerDied","Data":"c5e3a1bbadd7c4924073d1bbdb366e4974f808191f6e03bd23794da074574ce1"} Oct 30 19:16:50.392320 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.392290 5959 scope.go:122] "RemoveContainer" containerID="0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360" Oct 30 19:16:50.392700 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:50.392668 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360\": container with ID starting with 0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360 not found: ID does not exist" containerID="0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360" Oct 30 19:16:50.392834 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.392735 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360"} err="failed to get container status \"0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360\": rpc error: code = NotFound desc = could not find container \"0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360\": container with ID starting with 0f1b95483cd8f2de2e5532765a2899311bcf5fa15436fc8aeb8a7a645c33b360 not found: ID does not exist" Oct 30 19:16:50.392834 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.392766 5959 scope.go:122] "RemoveContainer" containerID="6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208" Oct 30 19:16:50.410341 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.410313 5959 scope.go:122] "RemoveContainer" containerID="6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208" Oct 30 19:16:50.410771 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:50.410734 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208\": container with ID starting with 6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208 not found: ID does not exist" containerID="6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208" Oct 30 19:16:50.410894 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.410774 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208"} err="failed to get container status \"6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208\": rpc error: code = NotFound desc = could not find container \"6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208\": container with ID starting with 6e02d58570729099813ee86e24e484b46d3b9dd50851156f5a1ece2dac302208 not found: ID does not exist" Oct 30 19:16:50.411691 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.411668 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq"] Oct 30 19:16:50.425982 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.420506 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-59hsq"] Oct 30 19:16:50.450482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.450426 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h"] Oct 30 19:16:50.453363 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:50.453327 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dxb8h"] Oct 30 19:16:51.798700 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:51.798661 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" path="/var/lib/kubelet/pods/aea10d0b-617b-4aa0-a73d-5d22a0e7153a/volumes" Oct 30 19:16:51.799337 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:51.799320 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" path="/var/lib/kubelet/pods/c9e4ec7d-7b73-42cc-94d3-5eb46d09872f/volumes" Oct 30 19:16:53.352527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:53.352489 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" podUID="cb7d1645-e7a7-433a-a39f-8d53f47d01ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:54.359291 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:54.359252 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" podUID="f2581e0a-b5b3-4df8-8b36-7f9b5232addc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:16:57.793425 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:57.793360 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:16:57.853542 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:57.853501 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj"] Oct 30 19:16:57.853801 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:57.853774 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" podUID="424643ce-2ec3-462d-82a8-f5a61ea7a329" containerName="maas-api" containerID="cri-o://dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612" gracePeriod=30 Oct 30 19:16:58.208151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.208115 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 19:16:58.354656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.354520 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-metrics-certs\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-metrics-certs\") pod \"424643ce-2ec3-462d-82a8-f5a61ea7a329\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " Oct 30 19:16:58.354656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.354587 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424643ce-2ec3-462d-82a8-f5a61ea7a329-kube-api-access-vbrqq\" (UniqueName: \"kubernetes.io/projected/424643ce-2ec3-462d-82a8-f5a61ea7a329-kube-api-access-vbrqq\") pod \"424643ce-2ec3-462d-82a8-f5a61ea7a329\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " Oct 30 19:16:58.354920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.354737 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424643ce-2ec3-462d-82a8-f5a61ea7a329-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424643ce-2ec3-462d-82a8-f5a61ea7a329-openshift-service-ca\") pod \"424643ce-2ec3-462d-82a8-f5a61ea7a329\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " Oct 30 19:16:58.355008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.354978 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-maas-api-tls\") pod \"424643ce-2ec3-462d-82a8-f5a61ea7a329\" (UID: \"424643ce-2ec3-462d-82a8-f5a61ea7a329\") " Oct 30 19:16:58.355086 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.355063 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424643ce-2ec3-462d-82a8-f5a61ea7a329-openshift-service-ca" pod "424643ce-2ec3-462d-82a8-f5a61ea7a329" (UID: "424643ce-2ec3-462d-82a8-f5a61ea7a329"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:58.355482 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.355459 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424643ce-2ec3-462d-82a8-f5a61ea7a329-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:58.357069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.357037 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-metrics-certs" pod "424643ce-2ec3-462d-82a8-f5a61ea7a329" (UID: "424643ce-2ec3-462d-82a8-f5a61ea7a329"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:58.357069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.357063 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-maas-api-tls" pod "424643ce-2ec3-462d-82a8-f5a61ea7a329" (UID: "424643ce-2ec3-462d-82a8-f5a61ea7a329"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:58.357210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.357162 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424643ce-2ec3-462d-82a8-f5a61ea7a329-kube-api-access-vbrqq" pod "424643ce-2ec3-462d-82a8-f5a61ea7a329" (UID: "424643ce-2ec3-462d-82a8-f5a61ea7a329"). InnerVolumeSpecName "kube-api-access-vbrqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:58.369150 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.369121 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff"] Oct 30 19:16:58.377143 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.375864 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs"] Oct 30 19:16:58.443572 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.443534 5959 generic.go:356] "Generic (PLEG): container finished" podID="424643ce-2ec3-462d-82a8-f5a61ea7a329" containerID="dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612" exitCode=0 Oct 30 19:16:58.443817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.443585 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" Oct 30 19:16:58.443817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.443651 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" event={"ID":"424643ce-2ec3-462d-82a8-f5a61ea7a329","Type":"ContainerDied","Data":"dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612"} Oct 30 19:16:58.443817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.443715 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj" event={"ID":"424643ce-2ec3-462d-82a8-f5a61ea7a329","Type":"ContainerDied","Data":"a58969f7f7ea43c61ab4351a5a1cbbdf9c34926ea84e80480503c6e619db05e4"} Oct 30 19:16:58.443817 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.443747 5959 scope.go:122] "RemoveContainer" containerID="dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612" Oct 30 19:16:58.456472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.456445 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbrqq\" (UniqueName: \"kubernetes.io/projected/424643ce-2ec3-462d-82a8-f5a61ea7a329-kube-api-access-vbrqq\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:58.456472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.456473 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:58.456714 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.456485 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424643ce-2ec3-462d-82a8-f5a61ea7a329-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:58.459416 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.459392 5959 scope.go:122] "RemoveContainer" containerID="dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612" Oct 30 19:16:58.459844 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:16:58.459818 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612\": container with ID starting with dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612 not found: ID does not exist" containerID="dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612" Oct 30 19:16:58.459939 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.459857 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612"} err="failed to get container status \"dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612\": rpc error: code = NotFound desc = could not find container \"dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612\": container with ID starting with dae2a8c3972691c75abefc10a8181baca2c9ccf273da2efaa717d42396df8612 not found: ID does not exist" Oct 30 19:16:58.477185 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.477142 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:16:58.480435 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.480401 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj"] Oct 30 19:16:58.487998 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:58.487970 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2889dd-57b5587478-flbqj"] Oct 30 19:16:59.800217 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:16:59.800176 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424643ce-2ec3-462d-82a8-f5a61ea7a329" path="/var/lib/kubelet/pods/424643ce-2ec3-462d-82a8-f5a61ea7a329/volumes" Oct 30 19:17:02.537665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:02.537628 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568"] Oct 30 19:17:02.538115 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:02.537894 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" containerID="cri-o://c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5" gracePeriod=30 Oct 30 19:17:02.553329 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:02.553294 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4"] Oct 30 19:17:02.553614 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:02.553556 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" containerID="cri-o://360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927" gracePeriod=30 Oct 30 19:17:02.563312 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:02.563279 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7"] Oct 30 19:17:02.563516 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:02.563497 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" containerID="cri-o://53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35" gracePeriod=30 Oct 30 19:17:03.066548 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:03.066509 5959 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:17:03.066773 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:03.066592 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:17:03.353378 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:03.353292 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" podUID="cb7d1645-e7a7-433a-a39f-8d53f47d01ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:17:03.803731 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:03.803698 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv"] Oct 30 19:17:04.359899 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:04.359861 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" podUID="f2581e0a-b5b3-4df8-8b36-7f9b5232addc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:17:05.493920 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.493881 5959 generic.go:356] "Generic (PLEG): container finished" podID="f2581e0a-b5b3-4df8-8b36-7f9b5232addc" containerID="dcce2ff4b1ba68c4e146a3c3506a3e40ae67262aabfd3c2dbbff95b37adb21c2" exitCode=0 Oct 30 19:17:05.494356 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.493984 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" event={"ID":"f2581e0a-b5b3-4df8-8b36-7f9b5232addc","Type":"ContainerDied","Data":"dcce2ff4b1ba68c4e146a3c3506a3e40ae67262aabfd3c2dbbff95b37adb21c2"} Oct 30 19:17:05.662004 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.661972 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:17:05.689223 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.689194 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:17:05.763646 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.763565 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-plugins-config-volume\") pod \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\" (UID: \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\") " Oct 30 19:17:05.763831 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.763792 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-kube-api-access-tb79l\" (UniqueName: \"kubernetes.io/projected/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-kube-api-access-tb79l\") pod \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\" (UID: \"cb7d1645-e7a7-433a-a39f-8d53f47d01ef\") " Oct 30 19:17:05.763957 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.763943 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-kube-api-access-vpfjr\" (UniqueName: \"kubernetes.io/projected/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-kube-api-access-vpfjr\") pod \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\" (UID: \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\") " Oct 30 19:17:05.764028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.763965 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-plugins-config-volume\") pod \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\" (UID: \"f2581e0a-b5b3-4df8-8b36-7f9b5232addc\") " Oct 30 19:17:05.764080 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.764025 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-plugins-config-volume" pod "cb7d1645-e7a7-433a-a39f-8d53f47d01ef" (UID: "cb7d1645-e7a7-433a-a39f-8d53f47d01ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:05.764417 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.764394 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-plugins-config-volume" pod "f2581e0a-b5b3-4df8-8b36-7f9b5232addc" (UID: "f2581e0a-b5b3-4df8-8b36-7f9b5232addc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:05.764417 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.764410 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:05.766329 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.766301 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-kube-api-access-tb79l" pod "cb7d1645-e7a7-433a-a39f-8d53f47d01ef" (UID: "cb7d1645-e7a7-433a-a39f-8d53f47d01ef"). InnerVolumeSpecName "kube-api-access-tb79l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:05.766452 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.766375 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-kube-api-access-vpfjr" pod "f2581e0a-b5b3-4df8-8b36-7f9b5232addc" (UID: "f2581e0a-b5b3-4df8-8b36-7f9b5232addc"). InnerVolumeSpecName "kube-api-access-vpfjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:05.865540 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.865489 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpfjr\" (UniqueName: \"kubernetes.io/projected/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-kube-api-access-vpfjr\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:05.865540 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.865536 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2581e0a-b5b3-4df8-8b36-7f9b5232addc-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:05.865911 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:05.865551 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb79l\" (UniqueName: \"kubernetes.io/projected/cb7d1645-e7a7-433a-a39f-8d53f47d01ef-kube-api-access-tb79l\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:06.307985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.307945 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4"] Oct 30 19:17:06.308261 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.308234 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" containerID="cri-o://c0d4f8b267ad7e800e72f8d65d5243ab7dcda505bed6af99b09b99aa44fe2463" gracePeriod=30 Oct 30 19:17:06.334921 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.334883 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p"] Oct 30 19:17:06.335175 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.335138 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" containerID="cri-o://0f22ea7bc924991a9f629da1b08552e22ef0b2f9c9818e4948c29b89d5f41d62" gracePeriod=30 Oct 30 19:17:06.502757 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.502721 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" event={"ID":"f2581e0a-b5b3-4df8-8b36-7f9b5232addc","Type":"ContainerDied","Data":"03f7e4c9b6fb97f9e6656385eed9ba9d96f2ed2f817151d71e9c6e4974179875"} Oct 30 19:17:06.503232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.502789 5959 scope.go:122] "RemoveContainer" containerID="dcce2ff4b1ba68c4e146a3c3506a3e40ae67262aabfd3c2dbbff95b37adb21c2" Oct 30 19:17:06.503232 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.502736 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6" Oct 30 19:17:06.504910 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.504883 5959 generic.go:356] "Generic (PLEG): container finished" podID="cb7d1645-e7a7-433a-a39f-8d53f47d01ef" containerID="926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140" exitCode=0 Oct 30 19:17:06.505040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.504944 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" event={"ID":"cb7d1645-e7a7-433a-a39f-8d53f47d01ef","Type":"ContainerDied","Data":"926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140"} Oct 30 19:17:06.505040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.505011 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" event={"ID":"cb7d1645-e7a7-433a-a39f-8d53f47d01ef","Type":"ContainerDied","Data":"f4d8bb2f4299c3c40aa63548d859a374803d9817f39068f4bbf62c37091d86eb"} Oct 30 19:17:06.505241 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.505225 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v" Oct 30 19:17:06.514210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.514187 5959 scope.go:122] "RemoveContainer" containerID="926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140" Oct 30 19:17:06.526728 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.526702 5959 scope.go:122] "RemoveContainer" containerID="926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140" Oct 30 19:17:06.527104 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:06.527081 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140\": container with ID starting with 926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140 not found: ID does not exist" containerID="926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140" Oct 30 19:17:06.527199 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.527119 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140"} err="failed to get container status \"926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140\": rpc error: code = NotFound desc = could not find container \"926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140\": container with ID starting with 926779bbd456e590730a8297de375cc7638f8089e2b48d7f2ae02860fdb1e140 not found: ID does not exist" Oct 30 19:17:06.534675 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.534644 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6"] Oct 30 19:17:06.544430 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.544399 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef6be7-b5b8d4c6d-cchx6"] Oct 30 19:17:06.558238 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.558144 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v"] Oct 30 19:17:06.566718 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:06.566666 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef6be7-b644cdf8c-7ts4v"] Oct 30 19:17:07.798474 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:07.798438 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7d1645-e7a7-433a-a39f-8d53f47d01ef" path="/var/lib/kubelet/pods/cb7d1645-e7a7-433a-a39f-8d53f47d01ef/volumes" Oct 30 19:17:07.799093 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:07.799074 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2581e0a-b5b3-4df8-8b36-7f9b5232addc" path="/var/lib/kubelet/pods/f2581e0a-b5b3-4df8-8b36-7f9b5232addc/volumes" Oct 30 19:17:07.862260 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:07.862230 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:17:07.909645 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:07.909579 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:17:07.913490 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:07.913466 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:17:08.007317 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007279 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-certs\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.007524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007347 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-istio-token\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-istio-token\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.007524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007383 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-credential-socket\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.007524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007419 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-token\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-token\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.007524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007453 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03100ce6-ad8c-4e96-8403-1da2674f46b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03100ce6-ad8c-4e96-8403-1da2674f46b5-istiod-ca-cert\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.007524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007478 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-data\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.007524 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007511 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-credential-socket\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007555 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cae719-52c3-492a-aa66-c4db3a011fee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cae719-52c3-492a-aa66-c4db3a011fee-istiod-ca-cert\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007600 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-certs\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007645 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-certs" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007715 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-socket\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007764 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-podinfo\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007767 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-credential-socket" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007808 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-kube-api-access-kkwlw\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-kube-api-access-kkwlw\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007841 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-credential-socket\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007868 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-envoy\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.007908 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007909 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-envoy\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007913 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-credential-socket" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007928 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-data" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007956 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-socket\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.007991 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8563f5fd-0739-44cb-acbf-e98721991c05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8563f5fd-0739-44cb-acbf-e98721991c05-istiod-ca-cert\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008013 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-certs" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008033 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-podinfo\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008034 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cae719-52c3-492a-aa66-c4db3a011fee-istiod-ca-cert" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008083 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-certs\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008100 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03100ce6-ad8c-4e96-8403-1da2674f46b5-istiod-ca-cert" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008121 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-token\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-token\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008188 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-envoy\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008218 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-kube-api-access-wgtpb\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-kube-api-access-wgtpb\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008221 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-credential-socket" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008248 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8563f5fd-0739-44cb-acbf-e98721991c05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8563f5fd-0739-44cb-acbf-e98721991c05-istio-podinfo\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008312 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-socket\") pod \"8563f5fd-0739-44cb-acbf-e98721991c05\" (UID: \"8563f5fd-0739-44cb-acbf-e98721991c05\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008346 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-data\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008395 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-kube-api-access-2rpw7\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-kube-api-access-2rpw7\") pod \"03100ce6-ad8c-4e96-8403-1da2674f46b5\" (UID: \"03100ce6-ad8c-4e96-8403-1da2674f46b5\") " Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008407 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-socket" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.008481 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008468 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-data\") pod \"e5cae719-52c3-492a-aa66-c4db3a011fee\" (UID: \"e5cae719-52c3-492a-aa66-c4db3a011fee\") " Oct 30 19:17:08.009471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008594 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-socket" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.009471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.008909 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8563f5fd-0739-44cb-acbf-e98721991c05-istiod-ca-cert" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:08.009471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009142 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-data" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.009471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009373 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009451 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009471 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009465 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03100ce6-ad8c-4e96-8403-1da2674f46b5-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009478 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009497 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-data" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009505 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009543 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cae719-52c3-492a-aa66-c4db3a011fee-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009577 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009593 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009635 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009651 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009666 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8563f5fd-0739-44cb-acbf-e98721991c05-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.009990 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.009885 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-socket" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.012121 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.010720 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-istio-token" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:08.012121 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.010828 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-kube-api-access-kkwlw" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "kube-api-access-kkwlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:08.012121 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.010851 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-certs" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.012121 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.011370 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-token" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:08.012463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.012372 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-podinfo" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:08.012463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.012449 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8563f5fd-0739-44cb-acbf-e98721991c05-istio-podinfo" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:08.012721 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.012673 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-kube-api-access-wgtpb" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "kube-api-access-wgtpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:08.012933 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.012909 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-envoy" pod "e5cae719-52c3-492a-aa66-c4db3a011fee" (UID: "e5cae719-52c3-492a-aa66-c4db3a011fee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.013224 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.013204 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-token" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:08.013324 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.013308 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-podinfo" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:08.013383 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.013320 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-envoy" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.013640 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.013594 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-kube-api-access-2rpw7" pod "03100ce6-ad8c-4e96-8403-1da2674f46b5" (UID: "03100ce6-ad8c-4e96-8403-1da2674f46b5"). InnerVolumeSpecName "kube-api-access-2rpw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:08.014077 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.014060 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-envoy" pod "8563f5fd-0739-44cb-acbf-e98721991c05" (UID: "8563f5fd-0739-44cb-acbf-e98721991c05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:08.111059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.110969 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111003 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111018 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111028 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkwlw\" (UniqueName: \"kubernetes.io/projected/8563f5fd-0739-44cb-acbf-e98721991c05-kube-api-access-kkwlw\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111037 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111046 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111053 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111059 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111062 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111070 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111081 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111095 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgtpb\" (UniqueName: \"kubernetes.io/projected/e5cae719-52c3-492a-aa66-c4db3a011fee-kube-api-access-wgtpb\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111104 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8563f5fd-0739-44cb-acbf-e98721991c05-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111115 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8563f5fd-0739-44cb-acbf-e98721991c05-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111123 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03100ce6-ad8c-4e96-8403-1da2674f46b5-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111133 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rpw7\" (UniqueName: \"kubernetes.io/projected/03100ce6-ad8c-4e96-8403-1da2674f46b5-kube-api-access-2rpw7\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.111448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.111142 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cae719-52c3-492a-aa66-c4db3a011fee-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.521344 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.521306 5959 generic.go:356] "Generic (PLEG): container finished" podID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerID="53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35" exitCode=0 Oct 30 19:17:08.521560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.521357 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" Oct 30 19:17:08.521560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.521386 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" event={"ID":"03100ce6-ad8c-4e96-8403-1da2674f46b5","Type":"ContainerDied","Data":"53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35"} Oct 30 19:17:08.521560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.521445 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7" event={"ID":"03100ce6-ad8c-4e96-8403-1da2674f46b5","Type":"ContainerDied","Data":"392b92f9f0e98c637152b193bbd6cd42c5f0300504b40d788ae96b267bbc3b7d"} Oct 30 19:17:08.521560 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.521471 5959 scope.go:122] "RemoveContainer" containerID="53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35" Oct 30 19:17:08.523194 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.522976 5959 generic.go:356] "Generic (PLEG): container finished" podID="8563f5fd-0739-44cb-acbf-e98721991c05" containerID="360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927" exitCode=0 Oct 30 19:17:08.523194 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.523020 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" Oct 30 19:17:08.523194 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.523070 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" event={"ID":"8563f5fd-0739-44cb-acbf-e98721991c05","Type":"ContainerDied","Data":"360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927"} Oct 30 19:17:08.523194 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.523124 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4" event={"ID":"8563f5fd-0739-44cb-acbf-e98721991c05","Type":"ContainerDied","Data":"0f649073be7c5b6d0b03fe5cad38be189b04f3c889db3bff00e42cbf7a55e854"} Oct 30 19:17:08.524843 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.524817 5959 generic.go:356] "Generic (PLEG): container finished" podID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerID="c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5" exitCode=0 Oct 30 19:17:08.524964 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.524873 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" Oct 30 19:17:08.525040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.524880 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" event={"ID":"e5cae719-52c3-492a-aa66-c4db3a011fee","Type":"ContainerDied","Data":"c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5"} Oct 30 19:17:08.525040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.525013 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568" event={"ID":"e5cae719-52c3-492a-aa66-c4db3a011fee","Type":"ContainerDied","Data":"bfa1b99d9e346523f3e89310daebeeff7efa79318056f9ea164809bbfdbc836e"} Oct 30 19:17:08.534751 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.534729 5959 scope.go:122] "RemoveContainer" containerID="53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35" Oct 30 19:17:08.535101 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:08.535079 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35\": container with ID starting with 53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35 not found: ID does not exist" containerID="53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35" Oct 30 19:17:08.535178 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.535110 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35"} err="failed to get container status \"53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35\": rpc error: code = NotFound desc = could not find container \"53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35\": container with ID starting with 53dc09f75b32b5a0a7dd7da655bf45fcd37bdb1644874b615e340ff5106d4d35 not found: ID does not exist" Oct 30 19:17:08.535178 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.535129 5959 scope.go:122] "RemoveContainer" containerID="360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927" Oct 30 19:17:08.545396 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.545374 5959 scope.go:122] "RemoveContainer" containerID="360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927" Oct 30 19:17:08.545739 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:08.545719 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927\": container with ID starting with 360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927 not found: ID does not exist" containerID="360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927" Oct 30 19:17:08.545809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.545747 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927"} err="failed to get container status \"360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927\": rpc error: code = NotFound desc = could not find container \"360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927\": container with ID starting with 360e418ec81ef9f9129d0ef00ad80ccda2f41e35e0829a740168e5e8e14c3927 not found: ID does not exist" Oct 30 19:17:08.545809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.545765 5959 scope.go:122] "RemoveContainer" containerID="c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5" Oct 30 19:17:08.554054 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.551228 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7"] Oct 30 19:17:08.554054 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.554008 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-h9sw7"] Oct 30 19:17:08.560428 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.560409 5959 scope.go:122] "RemoveContainer" containerID="c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5" Oct 30 19:17:08.560785 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:08.560761 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5\": container with ID starting with c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5 not found: ID does not exist" containerID="c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5" Oct 30 19:17:08.560874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.560796 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5"} err="failed to get container status \"c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5\": rpc error: code = NotFound desc = could not find container \"c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5\": container with ID starting with c57de6b6ee7d42fd4d4d752fb53039a15d9a6fb44e8a22836e21050de9c8b4f5 not found: ID does not exist" Oct 30 19:17:08.567092 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.566956 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568"] Oct 30 19:17:08.574655 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.574632 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-69568"] Oct 30 19:17:08.585874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.585848 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4"] Oct 30 19:17:08.589729 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.589706 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-9wjp4"] Oct 30 19:17:08.804641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.804528 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" podUID="108e4e52-68b1-458e-aca1-8706621e3a57" containerName="payload-processing" containerID="cri-o://276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6" gracePeriod=25 Oct 30 19:17:08.999203 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:08.999163 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" podUID="108e4e52-68b1-458e-aca1-8706621e3a57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:17:09.797869 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:09.797836 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" path="/var/lib/kubelet/pods/03100ce6-ad8c-4e96-8403-1da2674f46b5/volumes" Oct 30 19:17:09.798325 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:09.798311 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" path="/var/lib/kubelet/pods/8563f5fd-0739-44cb-acbf-e98721991c05/volumes" Oct 30 19:17:09.798836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:09.798822 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" path="/var/lib/kubelet/pods/e5cae719-52c3-492a-aa66-c4db3a011fee/volumes" Oct 30 19:17:11.569286 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.569246 5959 generic.go:356] "Generic (PLEG): container finished" podID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerID="c0d4f8b267ad7e800e72f8d65d5243ab7dcda505bed6af99b09b99aa44fe2463" exitCode=0 Oct 30 19:17:11.570012 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.569314 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" event={"ID":"a2606db6-8b96-4360-9519-6cda6ca1380f","Type":"ContainerDied","Data":"c0d4f8b267ad7e800e72f8d65d5243ab7dcda505bed6af99b09b99aa44fe2463"} Oct 30 19:17:11.571557 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.571529 5959 generic.go:356] "Generic (PLEG): container finished" podID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerID="0f22ea7bc924991a9f629da1b08552e22ef0b2f9c9818e4948c29b89d5f41d62" exitCode=0 Oct 30 19:17:11.571743 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.571637 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" event={"ID":"ea59e1b2-c9a5-4322-83f5-e8c1d9777880","Type":"ContainerDied","Data":"0f22ea7bc924991a9f629da1b08552e22ef0b2f9c9818e4948c29b89d5f41d62"} Oct 30 19:17:11.619797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.619769 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 19:17:11.684709 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.684684 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 19:17:11.764668 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764634 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-credential-socket\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.764883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764696 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-kube-api-access-xlf9r\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-kube-api-access-xlf9r\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.764883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764761 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2606db6-8b96-4360-9519-6cda6ca1380f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2606db6-8b96-4360-9519-6cda6ca1380f-istiod-ca-cert\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.764883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764813 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-token\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-token\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.764883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764837 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-certs\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.764883 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764881 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-envoy\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.765173 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764932 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-socket\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.765173 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764951 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-credential-socket" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.765173 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.764995 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-podinfo\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.765173 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.765126 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-data\") pod \"a2606db6-8b96-4360-9519-6cda6ca1380f\" (UID: \"a2606db6-8b96-4360-9519-6cda6ca1380f\") " Oct 30 19:17:11.765397 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.765171 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-certs" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.765397 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.765226 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2606db6-8b96-4360-9519-6cda6ca1380f-istiod-ca-cert" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:11.765756 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.765655 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-socket" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.765756 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.765662 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.765756 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.765699 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2606db6-8b96-4360-9519-6cda6ca1380f-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.765756 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.765714 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.765756 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.765763 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-data" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.767245 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.767223 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-kube-api-access-xlf9r" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "kube-api-access-xlf9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:11.767343 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.767248 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-envoy" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.767866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.767837 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-podinfo" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:11.767866 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.767840 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-token" pod "a2606db6-8b96-4360-9519-6cda6ca1380f" (UID: "a2606db6-8b96-4360-9519-6cda6ca1380f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:11.866437 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866354 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-podinfo\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866488 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-credential-socket\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866540 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-data\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866559 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-token\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-token\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866596 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-certs\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866672 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-envoy\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866695 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istiod-ca-cert\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866746 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-kube-api-access-6sn95\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-kube-api-access-6sn95\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866790 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-socket\") pod \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\" (UID: \"ea59e1b2-c9a5-4322-83f5-e8c1d9777880\") " Oct 30 19:17:11.866879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866848 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-credential-socket" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.866879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866863 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-data" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.867212 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.866965 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-certs" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.867212 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867201 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istiod-ca-cert" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:11.867422 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867399 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-socket" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.867710 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867666 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867710 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867702 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlf9r\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-kube-api-access-xlf9r\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867720 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867734 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867748 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867762 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867777 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867791 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2606db6-8b96-4360-9519-6cda6ca1380f-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867805 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2606db6-8b96-4360-9519-6cda6ca1380f-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867818 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.867900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.867831 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.869066 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.869041 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-podinfo" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:11.869066 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.869045 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-token" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:11.869187 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.869089 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-kube-api-access-6sn95" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "kube-api-access-6sn95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:11.869187 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.869091 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-envoy" pod "ea59e1b2-c9a5-4322-83f5-e8c1d9777880" (UID: "ea59e1b2-c9a5-4322-83f5-e8c1d9777880"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:11.968935 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.968903 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.968935 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.968930 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.968935 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.968941 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sn95\" (UniqueName: \"kubernetes.io/projected/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-kube-api-access-6sn95\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.969202 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:11.968950 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea59e1b2-c9a5-4322-83f5-e8c1d9777880-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.580935 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.580906 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" Oct 30 19:17:12.581428 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.580953 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4" event={"ID":"a2606db6-8b96-4360-9519-6cda6ca1380f","Type":"ContainerDied","Data":"0a613ebd3f1dd15747e3590a54659c684f8bc1f9a7eea12caa2dec32b53cdc7e"} Oct 30 19:17:12.581428 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.581018 5959 scope.go:122] "RemoveContainer" containerID="c0d4f8b267ad7e800e72f8d65d5243ab7dcda505bed6af99b09b99aa44fe2463" Oct 30 19:17:12.582730 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.582695 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" event={"ID":"ea59e1b2-c9a5-4322-83f5-e8c1d9777880","Type":"ContainerDied","Data":"80111b7ca3112502aa9278e64273ad1ac841d6a226f63763d528a7385c99e5a6"} Oct 30 19:17:12.582840 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.582726 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p" Oct 30 19:17:12.592223 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.592202 5959 scope.go:122] "RemoveContainer" containerID="0f22ea7bc924991a9f629da1b08552e22ef0b2f9c9818e4948c29b89d5f41d62" Oct 30 19:17:12.618344 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.618315 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4"] Oct 30 19:17:12.628391 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.628351 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-lhbs4"] Oct 30 19:17:12.657916 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.652281 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p"] Oct 30 19:17:12.657916 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:12.655156 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-g7n5p"] Oct 30 19:17:13.008527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.008482 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:17:13.317121 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.317030 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz"] Oct 30 19:17:13.349062 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.349001 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj"] Oct 30 19:17:13.396515 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.396463 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" containerID="cri-o://c8258316140d50ae03e54c5f30616be0e47b7a4c154272344fb91aa95b963dd7" gracePeriod=45 Oct 30 19:17:13.403352 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.403314 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" containerID="cri-o://5f8abfb09196767d59a4d43441ea2f96f56d5c66b241f3cd8e0493fb395e89b2" gracePeriod=45 Oct 30 19:17:13.593525 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.593497 5959 generic.go:356] "Generic (PLEG): container finished" podID="29b41146-9abf-4010-8c86-c473bbf18e19" containerID="c8258316140d50ae03e54c5f30616be0e47b7a4c154272344fb91aa95b963dd7" exitCode=0 Oct 30 19:17:13.593978 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.593619 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" event={"ID":"29b41146-9abf-4010-8c86-c473bbf18e19","Type":"ContainerDied","Data":"c8258316140d50ae03e54c5f30616be0e47b7a4c154272344fb91aa95b963dd7"} Oct 30 19:17:13.595208 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.595187 5959 generic.go:356] "Generic (PLEG): container finished" podID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerID="5f8abfb09196767d59a4d43441ea2f96f56d5c66b241f3cd8e0493fb395e89b2" exitCode=0 Oct 30 19:17:13.595278 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.595264 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" event={"ID":"3d9ce7f1-3bff-4808-ab73-d1281dd72f98","Type":"ContainerDied","Data":"5f8abfb09196767d59a4d43441ea2f96f56d5c66b241f3cd8e0493fb395e89b2"} Oct 30 19:17:13.654576 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.654535 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh"] Oct 30 19:17:13.654868 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.654843 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" podUID="e9633de6-8e2a-45ee-a960-5c6e797c384b" containerName="maas-api" containerID="cri-o://18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2" gracePeriod=30 Oct 30 19:17:13.761028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.760996 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:17:13.768592 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.768561 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:17:13.791103 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.791069 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b41146-9abf-4010-8c86-c473bbf18e19-kube-api-access-sh9cg\" (UniqueName: \"kubernetes.io/projected/29b41146-9abf-4010-8c86-c473bbf18e19-kube-api-access-sh9cg\") pod \"29b41146-9abf-4010-8c86-c473bbf18e19\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " Oct 30 19:17:13.791498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.791469 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tls-certs\") pod \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " Oct 30 19:17:13.792553 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.791844 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-home\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-home\") pod \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " Oct 30 19:17:13.792553 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.791910 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b41146-9abf-4010-8c86-c473bbf18e19-tls-certs\" (UniqueName: \"kubernetes.io/secret/29b41146-9abf-4010-8c86-c473bbf18e19-tls-certs\") pod \"29b41146-9abf-4010-8c86-c473bbf18e19\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " Oct 30 19:17:13.792553 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.791950 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-home\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-home\") pod \"29b41146-9abf-4010-8c86-c473bbf18e19\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " Oct 30 19:17:13.792553 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.792003 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-dshm\") pod \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " Oct 30 19:17:13.792553 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.792221 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-home" pod "3d9ce7f1-3bff-4808-ab73-d1281dd72f98" (UID: "3d9ce7f1-3bff-4808-ab73-d1281dd72f98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:13.792553 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.792312 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-home" pod "29b41146-9abf-4010-8c86-c473bbf18e19" (UID: "29b41146-9abf-4010-8c86-c473bbf18e19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:13.793705 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.792094 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-kube-api-access-zt499\" (UniqueName: \"kubernetes.io/projected/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-kube-api-access-zt499\") pod \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " Oct 30 19:17:13.793705 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.793295 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-dshm\") pod \"29b41146-9abf-4010-8c86-c473bbf18e19\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " Oct 30 19:17:13.793705 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.793408 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-model-cache\") pod \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " Oct 30 19:17:13.793705 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.793456 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-tmp-dir\") pod \"29b41146-9abf-4010-8c86-c473bbf18e19\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " Oct 30 19:17:13.793960 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.793857 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-model-cache" pod "3d9ce7f1-3bff-4808-ab73-d1281dd72f98" (UID: "3d9ce7f1-3bff-4808-ab73-d1281dd72f98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:13.793960 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.793949 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b41146-9abf-4010-8c86-c473bbf18e19-kube-api-access-sh9cg" pod "29b41146-9abf-4010-8c86-c473bbf18e19" (UID: "29b41146-9abf-4010-8c86-c473bbf18e19"). InnerVolumeSpecName "kube-api-access-sh9cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:13.794593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.794096 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-tmp-dir" pod "29b41146-9abf-4010-8c86-c473bbf18e19" (UID: "29b41146-9abf-4010-8c86-c473bbf18e19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:13.794593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.794490 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.794593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.794514 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.794593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.794530 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh9cg\" (UniqueName: \"kubernetes.io/projected/29b41146-9abf-4010-8c86-c473bbf18e19-kube-api-access-sh9cg\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.794593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.794547 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.794593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.794562 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.795379 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.795341 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-kube-api-access-zt499" pod "3d9ce7f1-3bff-4808-ab73-d1281dd72f98" (UID: "3d9ce7f1-3bff-4808-ab73-d1281dd72f98"). InnerVolumeSpecName "kube-api-access-zt499". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:13.797878 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.797814 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tls-certs" pod "3d9ce7f1-3bff-4808-ab73-d1281dd72f98" (UID: "3d9ce7f1-3bff-4808-ab73-d1281dd72f98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:13.798378 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.798353 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-dshm" pod "3d9ce7f1-3bff-4808-ab73-d1281dd72f98" (UID: "3d9ce7f1-3bff-4808-ab73-d1281dd72f98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:13.800217 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.799246 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-dshm" pod "29b41146-9abf-4010-8c86-c473bbf18e19" (UID: "29b41146-9abf-4010-8c86-c473bbf18e19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:13.804439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.802723 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b41146-9abf-4010-8c86-c473bbf18e19-tls-certs" pod "29b41146-9abf-4010-8c86-c473bbf18e19" (UID: "29b41146-9abf-4010-8c86-c473bbf18e19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:13.804439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.802948 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" path="/var/lib/kubelet/pods/a2606db6-8b96-4360-9519-6cda6ca1380f/volumes" Oct 30 19:17:13.804439 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.803469 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" path="/var/lib/kubelet/pods/ea59e1b2-c9a5-4322-83f5-e8c1d9777880/volumes" Oct 30 19:17:13.895527 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.895481 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tmp-dir\") pod \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\" (UID: \"3d9ce7f1-3bff-4808-ab73-d1281dd72f98\") " Oct 30 19:17:13.895752 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.895537 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-model-cache\") pod \"29b41146-9abf-4010-8c86-c473bbf18e19\" (UID: \"29b41146-9abf-4010-8c86-c473bbf18e19\") " Oct 30 19:17:13.895838 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.895782 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tmp-dir" pod "3d9ce7f1-3bff-4808-ab73-d1281dd72f98" (UID: "3d9ce7f1-3bff-4808-ab73-d1281dd72f98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:13.895903 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.895850 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-model-cache" pod "29b41146-9abf-4010-8c86-c473bbf18e19" (UID: "29b41146-9abf-4010-8c86-c473bbf18e19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:13.896112 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.896092 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt499\" (UniqueName: \"kubernetes.io/projected/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-kube-api-access-zt499\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.896162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.896115 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.896162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.896131 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.896162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.896141 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b41146-9abf-4010-8c86-c473bbf18e19-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.896162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.896149 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.896162 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.896157 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b41146-9abf-4010-8c86-c473bbf18e19-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.896316 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.896165 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9ce7f1-3bff-4808-ab73-d1281dd72f98-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:13.969392 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:13.969363 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:17:14.100962 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.100922 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9633de6-8e2a-45ee-a960-5c6e797c384b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9633de6-8e2a-45ee-a960-5c6e797c384b-openshift-service-ca\") pod \"e9633de6-8e2a-45ee-a960-5c6e797c384b\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " Oct 30 19:17:14.100962 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.100965 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9633de6-8e2a-45ee-a960-5c6e797c384b-kube-api-access-bb849\" (UniqueName: \"kubernetes.io/projected/e9633de6-8e2a-45ee-a960-5c6e797c384b-kube-api-access-bb849\") pod \"e9633de6-8e2a-45ee-a960-5c6e797c384b\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " Oct 30 19:17:14.101223 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.101095 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-metrics-certs\") pod \"e9633de6-8e2a-45ee-a960-5c6e797c384b\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " Oct 30 19:17:14.101223 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.101158 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls\") pod \"e9633de6-8e2a-45ee-a960-5c6e797c384b\" (UID: \"e9633de6-8e2a-45ee-a960-5c6e797c384b\") " Oct 30 19:17:14.101399 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.101374 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9633de6-8e2a-45ee-a960-5c6e797c384b-openshift-service-ca" pod "e9633de6-8e2a-45ee-a960-5c6e797c384b" (UID: "e9633de6-8e2a-45ee-a960-5c6e797c384b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:14.101797 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.101709 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9633de6-8e2a-45ee-a960-5c6e797c384b-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:14.104492 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.104195 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls" pod "e9633de6-8e2a-45ee-a960-5c6e797c384b" (UID: "e9633de6-8e2a-45ee-a960-5c6e797c384b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:14.104492 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.104230 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-metrics-certs" pod "e9633de6-8e2a-45ee-a960-5c6e797c384b" (UID: "e9633de6-8e2a-45ee-a960-5c6e797c384b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:14.104492 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.104277 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9633de6-8e2a-45ee-a960-5c6e797c384b-kube-api-access-bb849" pod "e9633de6-8e2a-45ee-a960-5c6e797c384b" (UID: "e9633de6-8e2a-45ee-a960-5c6e797c384b"). InnerVolumeSpecName "kube-api-access-bb849". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:14.203377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.203339 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb849\" (UniqueName: \"kubernetes.io/projected/e9633de6-8e2a-45ee-a960-5c6e797c384b-kube-api-access-bb849\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:14.203377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.203374 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:14.203377 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.203385 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9633de6-8e2a-45ee-a960-5c6e797c384b-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:14.604913 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.604884 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" Oct 30 19:17:14.605345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.604883 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff" event={"ID":"29b41146-9abf-4010-8c86-c473bbf18e19","Type":"ContainerDied","Data":"63832821f562df755bc41b7153f95193b84a973a97e0951c83da29cf4b271805"} Oct 30 19:17:14.605345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.605008 5959 scope.go:122] "RemoveContainer" containerID="c8258316140d50ae03e54c5f30616be0e47b7a4c154272344fb91aa95b963dd7" Oct 30 19:17:14.606390 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.606366 5959 generic.go:356] "Generic (PLEG): container finished" podID="e9633de6-8e2a-45ee-a960-5c6e797c384b" containerID="18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2" exitCode=0 Oct 30 19:17:14.606516 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.606435 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" Oct 30 19:17:14.606516 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.606438 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" event={"ID":"e9633de6-8e2a-45ee-a960-5c6e797c384b","Type":"ContainerDied","Data":"18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2"} Oct 30 19:17:14.606676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.606521 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh" event={"ID":"e9633de6-8e2a-45ee-a960-5c6e797c384b","Type":"ContainerDied","Data":"b1eaa7c694f90ed7eb1143480a12bc3d999ffef80b4f895540f2893aed4eaca0"} Oct 30 19:17:14.608321 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.608301 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" Oct 30 19:17:14.608321 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.608303 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs" event={"ID":"3d9ce7f1-3bff-4808-ab73-d1281dd72f98","Type":"ContainerDied","Data":"45e1d4cc0d9eaff352e6e215f75d61ecfaaeb09af941ff20dbc74fa55d2f5397"} Oct 30 19:17:14.617260 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.617243 5959 scope.go:122] "RemoveContainer" containerID="18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2" Oct 30 19:17:14.627352 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.627334 5959 scope.go:122] "RemoveContainer" containerID="18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2" Oct 30 19:17:14.627726 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:14.627706 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2\": container with ID starting with 18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2 not found: ID does not exist" containerID="18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2" Oct 30 19:17:14.627783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.627735 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2"} err="failed to get container status \"18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2\": rpc error: code = NotFound desc = could not find container \"18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2\": container with ID starting with 18707ab9c3cc7a1adcc8d79aa889e0b70a46d13d8f55ed3489dc6fc48fc8a5a2 not found: ID does not exist" Oct 30 19:17:14.627783 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.627753 5959 scope.go:122] "RemoveContainer" containerID="5f8abfb09196767d59a4d43441ea2f96f56d5c66b241f3cd8e0493fb395e89b2" Oct 30 19:17:14.669562 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.669481 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff"] Oct 30 19:17:14.677323 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.677295 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-2-w2-39bde3-kserve-67988bb58b-dn9ff"] Oct 30 19:17:14.706673 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.706638 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh"] Oct 30 19:17:14.722915 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.722888 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dcd326-587fd9fcc6-lgjwh"] Oct 30 19:17:14.745254 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.745218 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs"] Oct 30 19:17:14.760287 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:14.760251 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/distinct-w2-39bde3-kserve-84f56dcbcc-dc6hs"] Oct 30 19:17:15.796778 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:15.796738 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" path="/var/lib/kubelet/pods/29b41146-9abf-4010-8c86-c473bbf18e19/volumes" Oct 30 19:17:15.797219 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:15.797204 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" path="/var/lib/kubelet/pods/3d9ce7f1-3bff-4808-ab73-d1281dd72f98/volumes" Oct 30 19:17:15.797580 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:15.797568 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9633de6-8e2a-45ee-a960-5c6e797c384b" path="/var/lib/kubelet/pods/e9633de6-8e2a-45ee-a960-5c6e797c384b/volumes" Oct 30 19:17:16.833233 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:16.833180 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j"] Oct 30 19:17:16.833809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:16.833454 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" podUID="8de79c7d-5fbf-4bd1-9916-f696c5a07901" containerName="maas-api" containerID="cri-o://c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312" gracePeriod=30 Oct 30 19:17:17.156653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.156598 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 19:17:17.239591 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.239557 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de79c7d-5fbf-4bd1-9916-f696c5a07901-kube-api-access-59ddl\" (UniqueName: \"kubernetes.io/projected/8de79c7d-5fbf-4bd1-9916-f696c5a07901-kube-api-access-59ddl\") pod \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " Oct 30 19:17:17.239809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.239628 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de79c7d-5fbf-4bd1-9916-f696c5a07901-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de79c7d-5fbf-4bd1-9916-f696c5a07901-openshift-service-ca\") pod \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " Oct 30 19:17:17.239809 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.239710 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-maas-api-tls\") pod \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " Oct 30 19:17:17.239910 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.239824 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-metrics-certs\") pod \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\" (UID: \"8de79c7d-5fbf-4bd1-9916-f696c5a07901\") " Oct 30 19:17:17.240054 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.240026 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de79c7d-5fbf-4bd1-9916-f696c5a07901-openshift-service-ca" pod "8de79c7d-5fbf-4bd1-9916-f696c5a07901" (UID: "8de79c7d-5fbf-4bd1-9916-f696c5a07901"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:17.240351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.240277 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de79c7d-5fbf-4bd1-9916-f696c5a07901-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:17.242110 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.242080 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-maas-api-tls" pod "8de79c7d-5fbf-4bd1-9916-f696c5a07901" (UID: "8de79c7d-5fbf-4bd1-9916-f696c5a07901"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:17.242235 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.242110 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-metrics-certs" pod "8de79c7d-5fbf-4bd1-9916-f696c5a07901" (UID: "8de79c7d-5fbf-4bd1-9916-f696c5a07901"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:17.242235 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.242193 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de79c7d-5fbf-4bd1-9916-f696c5a07901-kube-api-access-59ddl" pod "8de79c7d-5fbf-4bd1-9916-f696c5a07901" (UID: "8de79c7d-5fbf-4bd1-9916-f696c5a07901"). InnerVolumeSpecName "kube-api-access-59ddl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:17.341242 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.341196 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:17.341242 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.341235 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59ddl\" (UniqueName: \"kubernetes.io/projected/8de79c7d-5fbf-4bd1-9916-f696c5a07901-kube-api-access-59ddl\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:17.341242 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.341252 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de79c7d-5fbf-4bd1-9916-f696c5a07901-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:17.646637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.642416 5959 generic.go:356] "Generic (PLEG): container finished" podID="8de79c7d-5fbf-4bd1-9916-f696c5a07901" containerID="c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312" exitCode=0 Oct 30 19:17:17.646637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.642716 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" event={"ID":"8de79c7d-5fbf-4bd1-9916-f696c5a07901","Type":"ContainerDied","Data":"c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312"} Oct 30 19:17:17.646637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.642799 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" event={"ID":"8de79c7d-5fbf-4bd1-9916-f696c5a07901","Type":"ContainerDied","Data":"cc72096df99df62b80e987710b0982be29acdbf447880d7eb75072a1578fd3a8"} Oct 30 19:17:17.646637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.642836 5959 scope.go:122] "RemoveContainer" containerID="c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312" Oct 30 19:17:17.646637 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.642992 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j" Oct 30 19:17:17.679588 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.679562 5959 scope.go:122] "RemoveContainer" containerID="c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312" Oct 30 19:17:17.680187 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.680158 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j"] Oct 30 19:17:17.680931 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:17.680486 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312\": container with ID starting with c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312 not found: ID does not exist" containerID="c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312" Oct 30 19:17:17.680931 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.680528 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312"} err="failed to get container status \"c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312\": rpc error: code = NotFound desc = could not find container \"c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312\": container with ID starting with c99e3b5b8c762f39066ce33df75cceb28bfb9d85a155a71cc2c35b7ddb507312 not found: ID does not exist" Oct 30 19:17:17.684578 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.684550 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-39bde3-6798b5d9b8-skr6j"] Oct 30 19:17:17.799729 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:17.799693 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de79c7d-5fbf-4bd1-9916-f696c5a07901" path="/var/lib/kubelet/pods/8de79c7d-5fbf-4bd1-9916-f696c5a07901/volumes" Oct 30 19:17:18.317673 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.317582 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" podUID="e4db960a-a125-48a3-a095-0cbc7d6b79b3" containerName="payload-processing" containerID="cri-o://f434c249255ee29c907e2b0935024903ab8e68667355e80f6154cbfb2aa0fd09" gracePeriod=25 Oct 30 19:17:18.350063 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.350020 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" podUID="ddb38225-6af5-4a9a-824a-2e232854b71a" containerName="payload-pre-processing" containerID="cri-o://f4ff0a5f28706481ce45aba8dd9b49703dcbd9f247664d3294aedb24118315e3" gracePeriod=25 Oct 30 19:17:18.672680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.668246 5959 generic.go:356] "Generic (PLEG): container finished" podID="e4db960a-a125-48a3-a095-0cbc7d6b79b3" containerID="f434c249255ee29c907e2b0935024903ab8e68667355e80f6154cbfb2aa0fd09" exitCode=0 Oct 30 19:17:18.672680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.668371 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" event={"ID":"e4db960a-a125-48a3-a095-0cbc7d6b79b3","Type":"ContainerDied","Data":"f434c249255ee29c907e2b0935024903ab8e68667355e80f6154cbfb2aa0fd09"} Oct 30 19:17:18.672680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.671110 5959 generic.go:356] "Generic (PLEG): container finished" podID="ddb38225-6af5-4a9a-824a-2e232854b71a" containerID="f4ff0a5f28706481ce45aba8dd9b49703dcbd9f247664d3294aedb24118315e3" exitCode=0 Oct 30 19:17:18.672680 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.671201 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" event={"ID":"ddb38225-6af5-4a9a-824a-2e232854b71a","Type":"ContainerDied","Data":"f4ff0a5f28706481ce45aba8dd9b49703dcbd9f247664d3294aedb24118315e3"} Oct 30 19:17:18.717201 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.717163 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:17:18.772659 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.772623 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb38225-6af5-4a9a-824a-2e232854b71a-kube-api-access-cj9sx\" (UniqueName: \"kubernetes.io/projected/ddb38225-6af5-4a9a-824a-2e232854b71a-kube-api-access-cj9sx\") pod \"ddb38225-6af5-4a9a-824a-2e232854b71a\" (UID: \"ddb38225-6af5-4a9a-824a-2e232854b71a\") " Oct 30 19:17:18.772853 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.772826 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb38225-6af5-4a9a-824a-2e232854b71a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb38225-6af5-4a9a-824a-2e232854b71a-plugins-config-volume\") pod \"ddb38225-6af5-4a9a-824a-2e232854b71a\" (UID: \"ddb38225-6af5-4a9a-824a-2e232854b71a\") " Oct 30 19:17:18.774160 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.773196 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb38225-6af5-4a9a-824a-2e232854b71a-plugins-config-volume" pod "ddb38225-6af5-4a9a-824a-2e232854b71a" (UID: "ddb38225-6af5-4a9a-824a-2e232854b71a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:18.774160 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.773505 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb38225-6af5-4a9a-824a-2e232854b71a-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:18.775463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.775429 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb38225-6af5-4a9a-824a-2e232854b71a-kube-api-access-cj9sx" pod "ddb38225-6af5-4a9a-824a-2e232854b71a" (UID: "ddb38225-6af5-4a9a-824a-2e232854b71a"). InnerVolumeSpecName "kube-api-access-cj9sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:18.781103 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.781080 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:17:18.874255 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.874222 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4db960a-a125-48a3-a095-0cbc7d6b79b3-kube-api-access-vthtn\" (UniqueName: \"kubernetes.io/projected/e4db960a-a125-48a3-a095-0cbc7d6b79b3-kube-api-access-vthtn\") pod \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\" (UID: \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\") " Oct 30 19:17:18.874438 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.874307 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4db960a-a125-48a3-a095-0cbc7d6b79b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4db960a-a125-48a3-a095-0cbc7d6b79b3-plugins-config-volume\") pod \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\" (UID: \"e4db960a-a125-48a3-a095-0cbc7d6b79b3\") " Oct 30 19:17:18.874746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.874721 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj9sx\" (UniqueName: \"kubernetes.io/projected/ddb38225-6af5-4a9a-824a-2e232854b71a-kube-api-access-cj9sx\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:18.874746 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.874728 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4db960a-a125-48a3-a095-0cbc7d6b79b3-plugins-config-volume" pod "e4db960a-a125-48a3-a095-0cbc7d6b79b3" (UID: "e4db960a-a125-48a3-a095-0cbc7d6b79b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:18.876808 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.876786 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4db960a-a125-48a3-a095-0cbc7d6b79b3-kube-api-access-vthtn" pod "e4db960a-a125-48a3-a095-0cbc7d6b79b3" (UID: "e4db960a-a125-48a3-a095-0cbc7d6b79b3"). InnerVolumeSpecName "kube-api-access-vthtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:18.976040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.975949 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4db960a-a125-48a3-a095-0cbc7d6b79b3-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:18.976040 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.975981 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vthtn\" (UniqueName: \"kubernetes.io/projected/e4db960a-a125-48a3-a095-0cbc7d6b79b3-kube-api-access-vthtn\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:18.998930 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:18.998891 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" podUID="108e4e52-68b1-458e-aca1-8706621e3a57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:17:19.679616 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.679579 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" Oct 30 19:17:19.680117 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.679597 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj" event={"ID":"ddb38225-6af5-4a9a-824a-2e232854b71a","Type":"ContainerDied","Data":"635176423f578167d27df554c0e5ab628536c7f91364ef64719278fe7c3545d6"} Oct 30 19:17:19.680117 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.679685 5959 scope.go:122] "RemoveContainer" containerID="f4ff0a5f28706481ce45aba8dd9b49703dcbd9f247664d3294aedb24118315e3" Oct 30 19:17:19.681204 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.681182 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" event={"ID":"e4db960a-a125-48a3-a095-0cbc7d6b79b3","Type":"ContainerDied","Data":"ad4ea73eb3ada52c94d0a40fb6fddf7b6b0312eb4befa3a10e98212bd326d267"} Oct 30 19:17:19.681309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.681217 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz" Oct 30 19:17:19.690514 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.690493 5959 scope.go:122] "RemoveContainer" containerID="f434c249255ee29c907e2b0935024903ab8e68667355e80f6154cbfb2aa0fd09" Oct 30 19:17:19.704782 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.704750 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj"] Oct 30 19:17:19.706618 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.706575 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dcd326-755bb5b555-7txnj"] Oct 30 19:17:19.717381 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.717351 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz"] Oct 30 19:17:19.721321 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.721292 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dcd326-76747774b6-v9wmz"] Oct 30 19:17:19.796463 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.796425 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb38225-6af5-4a9a-824a-2e232854b71a" path="/var/lib/kubelet/pods/ddb38225-6af5-4a9a-824a-2e232854b71a/volumes" Oct 30 19:17:19.796963 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:19.796942 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4db960a-a125-48a3-a095-0cbc7d6b79b3" path="/var/lib/kubelet/pods/e4db960a-a125-48a3-a095-0cbc7d6b79b3/volumes" Oct 30 19:17:22.037958 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:22.037917 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr"] Oct 30 19:17:22.042305 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:22.042253 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9593db2b-wz4dr"] Oct 30 19:17:23.798594 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:23.798558 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88531eb4-beff-48eb-b123-8da03308abca" path="/var/lib/kubelet/pods/88531eb4-beff-48eb-b123-8da03308abca/volumes" Oct 30 19:17:26.057132 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:26.057092 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5"] Oct 30 19:17:26.057765 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:26.057707 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" containerID="cri-o://9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105" gracePeriod=30 Oct 30 19:17:26.059371 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:26.059347 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l"] Oct 30 19:17:26.059643 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:26.059543 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" containerID="cri-o://ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8" gracePeriod=30 Oct 30 19:17:26.528365 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:26.528331 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2"] Oct 30 19:17:26.528578 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:26.528557 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" containerID="cri-o://b6277f4b41af3d2f8f0fd2eab8c6599a0d5cdd32ccbc314c84f28465d3f36434" gracePeriod=30 Oct 30 19:17:26.761893 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:26.761858 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:17:27.780230 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:27.780191 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:17:29.048337 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.048307 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 19:17:29.092719 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.092638 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108e4e52-68b1-458e-aca1-8706621e3a57-kube-api-access-klzrr\" (UniqueName: \"kubernetes.io/projected/108e4e52-68b1-458e-aca1-8706621e3a57-kube-api-access-klzrr\") pod \"108e4e52-68b1-458e-aca1-8706621e3a57\" (UID: \"108e4e52-68b1-458e-aca1-8706621e3a57\") " Oct 30 19:17:29.092719 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.092705 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108e4e52-68b1-458e-aca1-8706621e3a57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108e4e52-68b1-458e-aca1-8706621e3a57-plugins-config-volume\") pod \"108e4e52-68b1-458e-aca1-8706621e3a57\" (UID: \"108e4e52-68b1-458e-aca1-8706621e3a57\") " Oct 30 19:17:29.093097 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.093073 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108e4e52-68b1-458e-aca1-8706621e3a57-plugins-config-volume" pod "108e4e52-68b1-458e-aca1-8706621e3a57" (UID: "108e4e52-68b1-458e-aca1-8706621e3a57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:29.093283 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.093267 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108e4e52-68b1-458e-aca1-8706621e3a57-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:29.094952 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.094926 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108e4e52-68b1-458e-aca1-8706621e3a57-kube-api-access-klzrr" pod "108e4e52-68b1-458e-aca1-8706621e3a57" (UID: "108e4e52-68b1-458e-aca1-8706621e3a57"). InnerVolumeSpecName "kube-api-access-klzrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:29.194961 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.194926 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klzrr\" (UniqueName: \"kubernetes.io/projected/108e4e52-68b1-458e-aca1-8706621e3a57-kube-api-access-klzrr\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:29.741541 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.741505 5959 generic.go:356] "Generic (PLEG): container finished" podID="108e4e52-68b1-458e-aca1-8706621e3a57" containerID="276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6" exitCode=0 Oct 30 19:17:29.741875 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.741552 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" Oct 30 19:17:29.741875 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.741593 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" event={"ID":"108e4e52-68b1-458e-aca1-8706621e3a57","Type":"ContainerDied","Data":"276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6"} Oct 30 19:17:29.741875 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.741666 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" event={"ID":"108e4e52-68b1-458e-aca1-8706621e3a57","Type":"ContainerDied","Data":"1becb86c40af6831aed6bf194e11d9de27d8c8a6809cac2cef66e56ea958a29c"} Oct 30 19:17:29.741875 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.741690 5959 scope.go:122] "RemoveContainer" containerID="276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6" Oct 30 19:17:29.751355 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.751339 5959 scope.go:122] "RemoveContainer" containerID="276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6" Oct 30 19:17:29.751700 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:29.751680 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6\": container with ID starting with 276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6 not found: ID does not exist" containerID="276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6" Oct 30 19:17:29.751754 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.751710 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6"} err="failed to get container status \"276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6\": rpc error: code = NotFound desc = could not find container \"276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6\": container with ID starting with 276675051a5a5d8f45b9790a9edba8216d3bd7c05ba8851450c0563f18ae1ec6 not found: ID does not exist" Oct 30 19:17:29.763221 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.763197 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv"] Oct 30 19:17:29.766185 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.766163 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv"] Oct 30 19:17:29.798701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.798676 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108e4e52-68b1-458e-aca1-8706621e3a57" path="/var/lib/kubelet/pods/108e4e52-68b1-458e-aca1-8706621e3a57/volumes" Oct 30 19:17:29.998635 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:29.998525 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-39bde3-79487f87d-hvbtv" podUID="108e4e52-68b1-458e-aca1-8706621e3a57" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.46:50052\" within 1s: context deadline exceeded" Oct 30 19:17:29.998791 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:17:29.998652 5959 logging.go:55] [core] [Channel #5782 SubChannel #5783]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.46:50052", ServerName: "10.134.0.46:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.46:50052: operation was canceled" Oct 30 19:17:31.320153 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.320127 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 19:17:31.359182 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.359143 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 19:17:31.422803 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.422770 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-socket\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.422985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.422811 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-token\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-token\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.422985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.422846 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-socket\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.422985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.422880 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-credential-socket\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.422985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.422901 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-certs\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.422985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.422923 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-data\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.422985 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.422966 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-data\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.422994 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-kube-api-access-crxwt\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-kube-api-access-crxwt\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423032 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e0824e-05df-42c1-a737-90283fecef74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e0824e-05df-42c1-a737-90283fecef74-istiod-ca-cert\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423062 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-istio-token\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-istio-token\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423114 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-credential-socket\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423091 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-socket" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423167 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-certs\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423182 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-socket" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423199 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89e0824e-05df-42c1-a737-90283fecef74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e0824e-05df-42c1-a737-90283fecef74-istio-podinfo\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423238 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-certs" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423253 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-credential-socket" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423272 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-envoy\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.423318 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423302 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-data" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.423977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423362 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a468ed9d-e51f-422b-927d-abce4d0c7a36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a468ed9d-e51f-422b-927d-abce4d0c7a36-istiod-ca-cert\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.423977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423383 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-podinfo\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.423977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423422 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-envoy\") pod \"89e0824e-05df-42c1-a737-90283fecef74\" (UID: \"89e0824e-05df-42c1-a737-90283fecef74\") " Oct 30 19:17:31.423977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423455 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-kube-api-access-hfmcg\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-kube-api-access-hfmcg\") pod \"a468ed9d-e51f-422b-927d-abce4d0c7a36\" (UID: \"a468ed9d-e51f-422b-927d-abce4d0c7a36\") " Oct 30 19:17:31.423977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423567 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-credential-socket" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.423977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423694 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e0824e-05df-42c1-a737-90283fecef74-istiod-ca-cert" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:31.423977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423832 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-certs" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.423977 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.423957 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a468ed9d-e51f-422b-927d-abce4d0c7a36-istiod-ca-cert" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424094 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424125 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424140 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424154 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424167 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424180 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e0824e-05df-42c1-a737-90283fecef74-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424194 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424207 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.424511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424219 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a468ed9d-e51f-422b-927d-abce4d0c7a36-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.425168 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.424673 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-data" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.426470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.426298 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-kube-api-access-crxwt" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "kube-api-access-crxwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:31.426470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.426401 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-envoy" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.426470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.426413 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89e0824e-05df-42c1-a737-90283fecef74-istio-podinfo" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:31.426470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.426435 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-token" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:31.426470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.426465 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-podinfo" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:31.426871 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.426583 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-kube-api-access-hfmcg" pod "a468ed9d-e51f-422b-927d-abce4d0c7a36" (UID: "a468ed9d-e51f-422b-927d-abce4d0c7a36"). InnerVolumeSpecName "kube-api-access-hfmcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:31.427016 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.426996 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-envoy" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.427780 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.427762 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-istio-token" pod "89e0824e-05df-42c1-a737-90283fecef74" (UID: "89e0824e-05df-42c1-a737-90283fecef74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:31.525663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525594 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.525663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525654 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.525663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525665 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmcg\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-kube-api-access-hfmcg\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.525663 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525677 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.525938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525685 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e0824e-05df-42c1-a737-90283fecef74-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.525938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525694 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxwt\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-kube-api-access-crxwt\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.525938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525703 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e0824e-05df-42c1-a737-90283fecef74-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.525938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525711 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e0824e-05df-42c1-a737-90283fecef74-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.525938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.525718 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a468ed9d-e51f-422b-927d-abce4d0c7a36-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.758938 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.758905 5959 generic.go:356] "Generic (PLEG): container finished" podID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerID="9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105" exitCode=0 Oct 30 19:17:31.759115 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.758970 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" Oct 30 19:17:31.759115 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.758994 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" event={"ID":"a468ed9d-e51f-422b-927d-abce4d0c7a36","Type":"ContainerDied","Data":"9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105"} Oct 30 19:17:31.759115 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.759089 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5" event={"ID":"a468ed9d-e51f-422b-927d-abce4d0c7a36","Type":"ContainerDied","Data":"2487599d3056266032286e94239cc5cc1d5b8715d2cd70f3d4a3b69ab550ffa6"} Oct 30 19:17:31.759349 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.759122 5959 scope.go:122] "RemoveContainer" containerID="9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105" Oct 30 19:17:31.760491 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.760469 5959 generic.go:356] "Generic (PLEG): container finished" podID="89e0824e-05df-42c1-a737-90283fecef74" containerID="ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8" exitCode=0 Oct 30 19:17:31.760631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.760510 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" Oct 30 19:17:31.760631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.760558 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" event={"ID":"89e0824e-05df-42c1-a737-90283fecef74","Type":"ContainerDied","Data":"ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8"} Oct 30 19:17:31.760631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.760625 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l" event={"ID":"89e0824e-05df-42c1-a737-90283fecef74","Type":"ContainerDied","Data":"474cb89da8bd09b1acc5d51c50c409c99f5f8aec336cfd42019bb8b305133f75"} Oct 30 19:17:31.763012 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.762987 5959 generic.go:356] "Generic (PLEG): container finished" podID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerID="b6277f4b41af3d2f8f0fd2eab8c6599a0d5cdd32ccbc314c84f28465d3f36434" exitCode=0 Oct 30 19:17:31.763131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.763013 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" event={"ID":"3edfaee8-04db-4f64-bdec-efe3df8a5439","Type":"ContainerDied","Data":"b6277f4b41af3d2f8f0fd2eab8c6599a0d5cdd32ccbc314c84f28465d3f36434"} Oct 30 19:17:31.771309 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.771196 5959 scope.go:122] "RemoveContainer" containerID="9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105" Oct 30 19:17:31.772042 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:31.771996 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105\": container with ID starting with 9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105 not found: ID does not exist" containerID="9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105" Oct 30 19:17:31.772103 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.772050 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105"} err="failed to get container status \"9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105\": rpc error: code = NotFound desc = could not find container \"9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105\": container with ID starting with 9a2e9ff606dd2b52416a980404cb22a5bb9c9b9481faac41fa722764122c9105 not found: ID does not exist" Oct 30 19:17:31.772103 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.772072 5959 scope.go:122] "RemoveContainer" containerID="ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8" Oct 30 19:17:31.780995 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.780975 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:17:31.781788 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.781757 5959 scope.go:122] "RemoveContainer" containerID="ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8" Oct 30 19:17:31.782051 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:31.782033 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8\": container with ID starting with ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8 not found: ID does not exist" containerID="ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8" Oct 30 19:17:31.782112 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.782060 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8"} err="failed to get container status \"ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8\": rpc error: code = NotFound desc = could not find container \"ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8\": container with ID starting with ae9657899c55ccfa1295e522affcf56bcf7798b38a0c21d5ff240cdd7ac5bef8 not found: ID does not exist" Oct 30 19:17:31.788362 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.788337 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5"] Oct 30 19:17:31.798236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.798218 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-rx8d5"] Oct 30 19:17:31.806888 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.806870 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l"] Oct 30 19:17:31.817406 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.817378 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-4w65l"] Oct 30 19:17:31.828473 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828446 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edfaee8-04db-4f64-bdec-efe3df8a5439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edfaee8-04db-4f64-bdec-efe3df8a5439-istiod-ca-cert\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.828600 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828489 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-kube-api-access-8bqdg\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-kube-api-access-8bqdg\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.828600 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828524 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-credential-socket\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.828600 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828551 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-podinfo\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.828799 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828720 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-certs\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.829090 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828885 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-envoy\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.829090 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828923 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-data\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.829090 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828951 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-token\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-token\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.829090 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828960 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-credential-socket" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.829090 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828963 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edfaee8-04db-4f64-bdec-efe3df8a5439-istiod-ca-cert" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:31.829090 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.828994 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-socket\") pod \"3edfaee8-04db-4f64-bdec-efe3df8a5439\" (UID: \"3edfaee8-04db-4f64-bdec-efe3df8a5439\") " Oct 30 19:17:31.829472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.829250 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-socket" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.829472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.829268 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-data" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.829581 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.829547 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.829658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.829583 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.829658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.829599 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edfaee8-04db-4f64-bdec-efe3df8a5439-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.829658 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.829656 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.829923 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.829887 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-certs" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.831141 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.831119 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-podinfo" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:31.831570 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.831553 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-kube-api-access-8bqdg" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "kube-api-access-8bqdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:31.831724 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.831712 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-token" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:31.832010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.831995 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-envoy" pod "3edfaee8-04db-4f64-bdec-efe3df8a5439" (UID: "3edfaee8-04db-4f64-bdec-efe3df8a5439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:31.930664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.930536 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.930664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.930572 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.930664 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.930586 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.930902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.930666 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bqdg\" (UniqueName: \"kubernetes.io/projected/3edfaee8-04db-4f64-bdec-efe3df8a5439-kube-api-access-8bqdg\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:31.930902 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:31.930695 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edfaee8-04db-4f64-bdec-efe3df8a5439-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:32.772048 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:32.772017 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" Oct 30 19:17:32.772565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:32.772045 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2" event={"ID":"3edfaee8-04db-4f64-bdec-efe3df8a5439","Type":"ContainerDied","Data":"317e767913b40f528d52c2022bb9ba1caffb438a3d77e09903952ad04160d504"} Oct 30 19:17:32.772565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:32.772114 5959 scope.go:122] "RemoveContainer" containerID="b6277f4b41af3d2f8f0fd2eab8c6599a0d5cdd32ccbc314c84f28465d3f36434" Oct 30 19:17:32.799209 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:32.799182 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2"] Oct 30 19:17:32.811201 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:32.811176 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-zsbp2"] Oct 30 19:17:33.796505 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:33.796470 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" path="/var/lib/kubelet/pods/3edfaee8-04db-4f64-bdec-efe3df8a5439/volumes" Oct 30 19:17:33.797053 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:33.797033 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e0824e-05df-42c1-a737-90283fecef74" path="/var/lib/kubelet/pods/89e0824e-05df-42c1-a737-90283fecef74/volumes" Oct 30 19:17:33.797467 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:33.797454 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" path="/var/lib/kubelet/pods/a468ed9d-e51f-422b-927d-abce4d0c7a36/volumes" Oct 30 19:17:34.038234 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:34.038196 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89"] Oct 30 19:17:34.041063 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:34.041036 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea8e386-zzh89"] Oct 30 19:17:35.795869 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:35.795834 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0dad2ee-3398-4525-b9e4-fca02023f015" path="/var/lib/kubelet/pods/c0dad2ee-3398-4525-b9e4-fca02023f015/volumes" Oct 30 19:17:37.264552 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:37.264512 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw"] Oct 30 19:17:37.290299 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:37.290266 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx"] Oct 30 19:17:42.265434 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.265329 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" podUID="49ce5351-e16e-498b-a06d-8a70ca71768b" containerName="payload-processing" containerID="cri-o://8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8" gracePeriod=25 Oct 30 19:17:42.291005 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.290969 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" podUID="5eb94bfc-8707-4375-9f7f-5c454fcb06e2" containerName="payload-pre-processing" containerID="cri-o://f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d" gracePeriod=25 Oct 30 19:17:42.545814 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.545787 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:17:42.578952 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.578923 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:17:42.649976 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.649923 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ce5351-e16e-498b-a06d-8a70ca71768b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ce5351-e16e-498b-a06d-8a70ca71768b-plugins-config-volume\") pod \"49ce5351-e16e-498b-a06d-8a70ca71768b\" (UID: \"49ce5351-e16e-498b-a06d-8a70ca71768b\") " Oct 30 19:17:42.649976 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.649989 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ce5351-e16e-498b-a06d-8a70ca71768b-kube-api-access-rfqvv\" (UniqueName: \"kubernetes.io/projected/49ce5351-e16e-498b-a06d-8a70ca71768b-kube-api-access-rfqvv\") pod \"49ce5351-e16e-498b-a06d-8a70ca71768b\" (UID: \"49ce5351-e16e-498b-a06d-8a70ca71768b\") " Oct 30 19:17:42.650427 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.650397 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ce5351-e16e-498b-a06d-8a70ca71768b-plugins-config-volume" pod "49ce5351-e16e-498b-a06d-8a70ca71768b" (UID: "49ce5351-e16e-498b-a06d-8a70ca71768b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:42.652358 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.652331 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ce5351-e16e-498b-a06d-8a70ca71768b-kube-api-access-rfqvv" pod "49ce5351-e16e-498b-a06d-8a70ca71768b" (UID: "49ce5351-e16e-498b-a06d-8a70ca71768b"). InnerVolumeSpecName "kube-api-access-rfqvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:42.750794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.750749 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-kube-api-access-hgj2b\" (UniqueName: \"kubernetes.io/projected/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-kube-api-access-hgj2b\") pod \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\" (UID: \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\") " Oct 30 19:17:42.750989 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.750969 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-plugins-config-volume\") pod \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\" (UID: \"5eb94bfc-8707-4375-9f7f-5c454fcb06e2\") " Oct 30 19:17:42.751210 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.751193 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ce5351-e16e-498b-a06d-8a70ca71768b-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.751303 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.751213 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfqvv\" (UniqueName: \"kubernetes.io/projected/49ce5351-e16e-498b-a06d-8a70ca71768b-kube-api-access-rfqvv\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.751374 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.751345 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-plugins-config-volume" pod "5eb94bfc-8707-4375-9f7f-5c454fcb06e2" (UID: "5eb94bfc-8707-4375-9f7f-5c454fcb06e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:42.752971 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.752949 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-kube-api-access-hgj2b" pod "5eb94bfc-8707-4375-9f7f-5c454fcb06e2" (UID: "5eb94bfc-8707-4375-9f7f-5c454fcb06e2"). InnerVolumeSpecName "kube-api-access-hgj2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:42.828531 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.828434 5959 generic.go:356] "Generic (PLEG): container finished" podID="49ce5351-e16e-498b-a06d-8a70ca71768b" containerID="8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8" exitCode=0 Oct 30 19:17:42.828531 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.828485 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" Oct 30 19:17:42.828531 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.828511 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" event={"ID":"49ce5351-e16e-498b-a06d-8a70ca71768b","Type":"ContainerDied","Data":"8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8"} Oct 30 19:17:42.828853 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.828563 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw" event={"ID":"49ce5351-e16e-498b-a06d-8a70ca71768b","Type":"ContainerDied","Data":"8ad0cfba0c44a751e1fa703b3b1d94a580d9c3ce07ae2a00c9b2a54726da54c7"} Oct 30 19:17:42.828853 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.828585 5959 scope.go:122] "RemoveContainer" containerID="8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8" Oct 30 19:17:42.830470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.830439 5959 generic.go:356] "Generic (PLEG): container finished" podID="5eb94bfc-8707-4375-9f7f-5c454fcb06e2" containerID="f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d" exitCode=0 Oct 30 19:17:42.830579 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.830474 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" event={"ID":"5eb94bfc-8707-4375-9f7f-5c454fcb06e2","Type":"ContainerDied","Data":"f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d"} Oct 30 19:17:42.830579 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.830517 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" Oct 30 19:17:42.830710 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.830531 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx" event={"ID":"5eb94bfc-8707-4375-9f7f-5c454fcb06e2","Type":"ContainerDied","Data":"dd9fd2f27bc4a0ae0624be7af58ad01a04a92f10be8e2c22ab9cbfea59668594"} Oct 30 19:17:42.845662 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.845640 5959 scope.go:122] "RemoveContainer" containerID="8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8" Oct 30 19:17:42.850011 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:42.848364 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8\": container with ID starting with 8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8 not found: ID does not exist" containerID="8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8" Oct 30 19:17:42.850011 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.848421 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8"} err="failed to get container status \"8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8\": rpc error: code = NotFound desc = could not find container \"8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8\": container with ID starting with 8142297085f4ee1313ab3e225f98066c363c7ea7b4ccf08df4f380a7fc4281a8 not found: ID does not exist" Oct 30 19:17:42.850011 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.848448 5959 scope.go:122] "RemoveContainer" containerID="f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d" Oct 30 19:17:42.851865 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.851843 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgj2b\" (UniqueName: \"kubernetes.io/projected/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-kube-api-access-hgj2b\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.851952 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.851867 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb94bfc-8707-4375-9f7f-5c454fcb06e2-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.860022 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.859999 5959 scope.go:122] "RemoveContainer" containerID="f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d" Oct 30 19:17:42.860357 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:42.860330 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d\": container with ID starting with f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d not found: ID does not exist" containerID="f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d" Oct 30 19:17:42.860424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.860371 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d"} err="failed to get container status \"f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d\": rpc error: code = NotFound desc = could not find container \"f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d\": container with ID starting with f8a15f8213fcfc3ae5070a64c93770fb38a57c2f0d5b5eabc8faca4d58a39b2d not found: ID does not exist" Oct 30 19:17:42.860424 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.860417 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx"] Oct 30 19:17:42.863940 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.863910 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-46385c-766578f7d7-q5wjx"] Oct 30 19:17:42.874019 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.873986 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw"] Oct 30 19:17:42.875975 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.875949 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-46385c-77fc5b7bb6-lzlpw"] Oct 30 19:17:42.962899 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:42.962858 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:17:43.795631 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:43.795571 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ce5351-e16e-498b-a06d-8a70ca71768b" path="/var/lib/kubelet/pods/49ce5351-e16e-498b-a06d-8a70ca71768b/volumes" Oct 30 19:17:43.796079 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:43.796062 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb94bfc-8707-4375-9f7f-5c454fcb06e2" path="/var/lib/kubelet/pods/5eb94bfc-8707-4375-9f7f-5c454fcb06e2/volumes" Oct 30 19:17:52.679260 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:52.679214 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj"] Oct 30 19:17:52.696585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:52.696550 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98"] Oct 30 19:17:57.779780 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:17:57.779738 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:17:58.004041 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:17:58.003992 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x"] Oct 30 19:18:03.004656 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:03.004564 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" podUID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" containerName="payload-pre-processing" containerID="cri-o://bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab" gracePeriod=25 Oct 30 19:18:03.049749 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:03.049705 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" podUID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:18:07.701442 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.701381 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" containerID="cri-o://ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8" gracePeriod=45 Oct 30 19:18:07.718470 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.718436 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" containerID="cri-o://ec8f48c7d9fb12061d5d9492a8e5f78e81d5dae2942226af437d25385ec49ad9" gracePeriod=45 Oct 30 19:18:07.972288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.972254 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:18:07.975562 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.975499 5959 generic.go:356] "Generic (PLEG): container finished" podID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerID="ec8f48c7d9fb12061d5d9492a8e5f78e81d5dae2942226af437d25385ec49ad9" exitCode=0 Oct 30 19:18:07.975796 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.975570 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" event={"ID":"1f3860e2-becb-4660-9d93-ff8c50373fb7","Type":"ContainerDied","Data":"ec8f48c7d9fb12061d5d9492a8e5f78e81d5dae2942226af437d25385ec49ad9"} Oct 30 19:18:07.990735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.979546 5959 generic.go:356] "Generic (PLEG): container finished" podID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerID="ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8" exitCode=0 Oct 30 19:18:07.990735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.979685 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" event={"ID":"96ba3eb5-1b12-4e2b-9fec-9842167e514a","Type":"ContainerDied","Data":"ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8"} Oct 30 19:18:07.990735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.979732 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" event={"ID":"96ba3eb5-1b12-4e2b-9fec-9842167e514a","Type":"ContainerDied","Data":"3ead26c670601e0f115a80c32527b8467e3db81dbb8c1a5c741d46dea70be6d6"} Oct 30 19:18:07.990735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.979789 5959 scope.go:122] "RemoveContainer" containerID="ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8" Oct 30 19:18:07.990735 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:07.980033 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" Oct 30 19:18:08.024122 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.024093 5959 scope.go:122] "RemoveContainer" containerID="ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8" Oct 30 19:18:08.024496 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:18:08.024459 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8\": container with ID starting with ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8 not found: ID does not exist" containerID="ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8" Oct 30 19:18:08.024616 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.024507 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8"} err="failed to get container status \"ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8\": rpc error: code = NotFound desc = could not find container \"ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8\": container with ID starting with ae023ee61e5dbe1d9db433241be8249b071a068eee16347f03ab00fa9ea973a8 not found: ID does not exist" Oct 30 19:18:08.040072 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.040035 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:18:08.122928 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.122875 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ba3eb5-1b12-4e2b-9fec-9842167e514a-kube-api-access-58fms\" (UniqueName: \"kubernetes.io/projected/96ba3eb5-1b12-4e2b-9fec-9842167e514a-kube-api-access-58fms\") pod \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " Oct 30 19:18:08.123140 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.122978 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-dshm\") pod \"1f3860e2-becb-4660-9d93-ff8c50373fb7\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " Oct 30 19:18:08.123140 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123005 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-tmp-dir\") pod \"1f3860e2-becb-4660-9d93-ff8c50373fb7\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " Oct 30 19:18:08.123140 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123087 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-dshm\") pod \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " Oct 30 19:18:08.123351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123179 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tls-certs\" (UniqueName: \"kubernetes.io/secret/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tls-certs\") pod \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " Oct 30 19:18:08.123351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123233 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3860e2-becb-4660-9d93-ff8c50373fb7-kube-api-access-dvtdf\" (UniqueName: \"kubernetes.io/projected/1f3860e2-becb-4660-9d93-ff8c50373fb7-kube-api-access-dvtdf\") pod \"1f3860e2-becb-4660-9d93-ff8c50373fb7\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " Oct 30 19:18:08.123351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123257 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3860e2-becb-4660-9d93-ff8c50373fb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3860e2-becb-4660-9d93-ff8c50373fb7-tls-certs\") pod \"1f3860e2-becb-4660-9d93-ff8c50373fb7\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " Oct 30 19:18:08.123351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123301 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-tmp-dir" pod "1f3860e2-becb-4660-9d93-ff8c50373fb7" (UID: "1f3860e2-becb-4660-9d93-ff8c50373fb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.123351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123322 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-home\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-home\") pod \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " Oct 30 19:18:08.123653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123381 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-model-cache\") pod \"1f3860e2-becb-4660-9d93-ff8c50373fb7\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " Oct 30 19:18:08.123653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123417 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-home\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-home\") pod \"1f3860e2-becb-4660-9d93-ff8c50373fb7\" (UID: \"1f3860e2-becb-4660-9d93-ff8c50373fb7\") " Oct 30 19:18:08.123653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123450 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-model-cache\") pod \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " Oct 30 19:18:08.123653 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.123529 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tmp-dir\") pod \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\" (UID: \"96ba3eb5-1b12-4e2b-9fec-9842167e514a\") " Oct 30 19:18:08.124567 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.124191 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.124567 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.124190 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-home" pod "96ba3eb5-1b12-4e2b-9fec-9842167e514a" (UID: "96ba3eb5-1b12-4e2b-9fec-9842167e514a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.124567 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.124512 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-model-cache" pod "1f3860e2-becb-4660-9d93-ff8c50373fb7" (UID: "1f3860e2-becb-4660-9d93-ff8c50373fb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.124567 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.124501 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-model-cache" pod "96ba3eb5-1b12-4e2b-9fec-9842167e514a" (UID: "96ba3eb5-1b12-4e2b-9fec-9842167e514a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.124896 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.124575 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tmp-dir" pod "96ba3eb5-1b12-4e2b-9fec-9842167e514a" (UID: "96ba3eb5-1b12-4e2b-9fec-9842167e514a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.124896 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.124586 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-home" pod "1f3860e2-becb-4660-9d93-ff8c50373fb7" (UID: "1f3860e2-becb-4660-9d93-ff8c50373fb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.126319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.126291 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-dshm" pod "1f3860e2-becb-4660-9d93-ff8c50373fb7" (UID: "1f3860e2-becb-4660-9d93-ff8c50373fb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.126319 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.126302 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tls-certs" pod "96ba3eb5-1b12-4e2b-9fec-9842167e514a" (UID: "96ba3eb5-1b12-4e2b-9fec-9842167e514a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:08.126494 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.126410 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3860e2-becb-4660-9d93-ff8c50373fb7-kube-api-access-dvtdf" pod "1f3860e2-becb-4660-9d93-ff8c50373fb7" (UID: "1f3860e2-becb-4660-9d93-ff8c50373fb7"). InnerVolumeSpecName "kube-api-access-dvtdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:08.126625 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.126570 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ba3eb5-1b12-4e2b-9fec-9842167e514a-kube-api-access-58fms" pod "96ba3eb5-1b12-4e2b-9fec-9842167e514a" (UID: "96ba3eb5-1b12-4e2b-9fec-9842167e514a"). InnerVolumeSpecName "kube-api-access-58fms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:08.126739 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.126663 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3860e2-becb-4660-9d93-ff8c50373fb7-tls-certs" pod "1f3860e2-becb-4660-9d93-ff8c50373fb7" (UID: "1f3860e2-becb-4660-9d93-ff8c50373fb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:08.126739 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.126729 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-dshm" pod "96ba3eb5-1b12-4e2b-9fec-9842167e514a" (UID: "96ba3eb5-1b12-4e2b-9fec-9842167e514a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.225764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225664 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.225764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225697 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.225764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225710 5959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tmp-dir\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.225764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225726 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58fms\" (UniqueName: \"kubernetes.io/projected/96ba3eb5-1b12-4e2b-9fec-9842167e514a-kube-api-access-58fms\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.225764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225739 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.225764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225749 5959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-dshm\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.225764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225760 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96ba3eb5-1b12-4e2b-9fec-9842167e514a-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.225764 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225771 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvtdf\" (UniqueName: \"kubernetes.io/projected/1f3860e2-becb-4660-9d93-ff8c50373fb7-kube-api-access-dvtdf\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.226448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225783 5959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3860e2-becb-4660-9d93-ff8c50373fb7-tls-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.226448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225793 5959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96ba3eb5-1b12-4e2b-9fec-9842167e514a-home\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.226448 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.225803 5959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3860e2-becb-4660-9d93-ff8c50373fb7-model-cache\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.320003 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.319968 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj"] Oct 30 19:18:08.325821 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.325794 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj"] Oct 30 19:18:08.916038 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.915991 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/distinct-w1-85cb41-kserve-674dc75c77-tfcjj" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:18:08.985727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.985598 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" event={"ID":"1f3860e2-becb-4660-9d93-ff8c50373fb7","Type":"ContainerDied","Data":"fbbd59b5a3d5140860a9549897a1c4c604c676c35aea42b4eca45f9798a1fe17"} Oct 30 19:18:08.985727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.985679 5959 scope.go:122] "RemoveContainer" containerID="ec8f48c7d9fb12061d5d9492a8e5f78e81d5dae2942226af437d25385ec49ad9" Oct 30 19:18:08.985727 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:08.985630 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98" Oct 30 19:18:09.008454 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:09.008425 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98"] Oct 30 19:18:09.013292 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:09.013265 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/simulator-w1-85cb41-kserve-7c986d86bc-6gd98"] Oct 30 19:18:09.017080 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:18:09.017054 5959 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f3860e2_becb_4660_9d93_ff8c50373fb7.slice/crio-fbbd59b5a3d5140860a9549897a1c4c604c676c35aea42b4eca45f9798a1fe17\": RecentStats: unable to find data in memory cache]" Oct 30 19:18:09.797584 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:09.797551 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" path="/var/lib/kubelet/pods/1f3860e2-becb-4660-9d93-ff8c50373fb7/volumes" Oct 30 19:18:09.798028 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:09.798013 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" path="/var/lib/kubelet/pods/96ba3eb5-1b12-4e2b-9fec-9842167e514a/volumes" Oct 30 19:18:10.875010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:10.874979 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:18:13.050383 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:13.050344 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" podUID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:18:13.689429 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:13.689210 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk"] Oct 30 19:18:13.700159 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:13.698420 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" podUID="cd779df3-489c-491c-8f35-747a9536e4dd" containerName="maas-api" containerID="cri-o://c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4" gracePeriod=30 Oct 30 19:18:13.964650 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:13.964622 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 19:18:14.020419 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.020379 5959 generic.go:356] "Generic (PLEG): container finished" podID="cd779df3-489c-491c-8f35-747a9536e4dd" containerID="c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4" exitCode=0 Oct 30 19:18:14.020623 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.020434 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" Oct 30 19:18:14.020623 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.020449 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" event={"ID":"cd779df3-489c-491c-8f35-747a9536e4dd","Type":"ContainerDied","Data":"c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4"} Oct 30 19:18:14.020623 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.020507 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk" event={"ID":"cd779df3-489c-491c-8f35-747a9536e4dd","Type":"ContainerDied","Data":"bfffe584936f2310abfecae282d9ce61e2ae25b4e16983eaca9b6d9931ea012d"} Oct 30 19:18:14.020623 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.020538 5959 scope.go:122] "RemoveContainer" containerID="c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4" Oct 30 19:18:14.030874 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.030847 5959 scope.go:122] "RemoveContainer" containerID="c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4" Oct 30 19:18:14.031277 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:18:14.031255 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4\": container with ID starting with c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4 not found: ID does not exist" containerID="c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4" Oct 30 19:18:14.031345 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.031288 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4"} err="failed to get container status \"c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4\": rpc error: code = NotFound desc = could not find container \"c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4\": container with ID starting with c68a294a15ad7492fe364e5646f07acff90a385ec7807bd1b6db01534bd63de4 not found: ID does not exist" Oct 30 19:18:14.090102 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.090071 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-maas-api-tls\") pod \"cd779df3-489c-491c-8f35-747a9536e4dd\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " Oct 30 19:18:14.090585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.090219 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd779df3-489c-491c-8f35-747a9536e4dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd779df3-489c-491c-8f35-747a9536e4dd-openshift-service-ca\") pod \"cd779df3-489c-491c-8f35-747a9536e4dd\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " Oct 30 19:18:14.090585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.090248 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-metrics-certs\") pod \"cd779df3-489c-491c-8f35-747a9536e4dd\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " Oct 30 19:18:14.090585 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.090272 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd779df3-489c-491c-8f35-747a9536e4dd-kube-api-access-h9qvb\" (UniqueName: \"kubernetes.io/projected/cd779df3-489c-491c-8f35-747a9536e4dd-kube-api-access-h9qvb\") pod \"cd779df3-489c-491c-8f35-747a9536e4dd\" (UID: \"cd779df3-489c-491c-8f35-747a9536e4dd\") " Oct 30 19:18:14.090802 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.090640 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd779df3-489c-491c-8f35-747a9536e4dd-openshift-service-ca" pod "cd779df3-489c-491c-8f35-747a9536e4dd" (UID: "cd779df3-489c-491c-8f35-747a9536e4dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:14.092967 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.092937 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-maas-api-tls" pod "cd779df3-489c-491c-8f35-747a9536e4dd" (UID: "cd779df3-489c-491c-8f35-747a9536e4dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:14.093078 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.092995 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-metrics-certs" pod "cd779df3-489c-491c-8f35-747a9536e4dd" (UID: "cd779df3-489c-491c-8f35-747a9536e4dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:14.093166 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.093140 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd779df3-489c-491c-8f35-747a9536e4dd-kube-api-access-h9qvb" pod "cd779df3-489c-491c-8f35-747a9536e4dd" (UID: "cd779df3-489c-491c-8f35-747a9536e4dd"). InnerVolumeSpecName "kube-api-access-h9qvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:14.192151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.192061 5959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd779df3-489c-491c-8f35-747a9536e4dd-openshift-service-ca\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:14.192151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.192098 5959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-metrics-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:14.192151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.192108 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9qvb\" (UniqueName: \"kubernetes.io/projected/cd779df3-489c-491c-8f35-747a9536e4dd-kube-api-access-h9qvb\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:14.192151 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.192118 5959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd779df3-489c-491c-8f35-747a9536e4dd-maas-api-tls\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:14.342479 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.342440 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk"] Oct 30 19:18:14.345955 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:14.345929 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-85cb41-86d689cb56-s5dmk"] Oct 30 19:18:15.795701 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:15.795669 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd779df3-489c-491c-8f35-747a9536e4dd" path="/var/lib/kubelet/pods/cd779df3-489c-491c-8f35-747a9536e4dd/volumes" Oct 30 19:18:16.142075 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:16.142044 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-d9fgc"] Oct 30 19:18:16.588216 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:16.588131 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk"] Oct 30 19:18:16.588388 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:16.588360 5959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" containerID="cri-o://88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b" gracePeriod=30 Oct 30 19:18:21.844464 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.844439 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 19:18:21.884156 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884077 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-kube-api-access-988rz\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-kube-api-access-988rz\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.884328 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884174 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-socket\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.884328 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884251 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-credential-socket\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.884328 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884285 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-envoy\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.884451 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884350 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-podinfo\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.884451 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884403 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istiod-ca-cert\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.884547 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884479 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-socket" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.884622 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884550 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-credential-socket" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.884881 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884855 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-token\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-token\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.884881 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884861 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istiod-ca-cert" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:21.885241 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.884927 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-data\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.885241 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.885032 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-certs\") pod \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\" (UID: \"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e\") " Oct 30 19:18:21.885363 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.885256 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-data" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.885498 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.885476 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-certs" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.885593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.885496 5959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istiod-ca-cert\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.885593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.885518 5959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-data\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.885593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.885531 5959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.885593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.885546 5959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-credential-socket\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.886644 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.886596 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-kube-api-access-988rz" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "kube-api-access-988rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:21.887251 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.887199 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-envoy" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.887347 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.887249 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-podinfo" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:18:21.887494 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.887476 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-token" pod "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" (UID: "5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:21.986565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.986528 5959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-token\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.986565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.986559 5959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-workload-certs\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.986565 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.986573 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-988rz\" (UniqueName: \"kubernetes.io/projected/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-kube-api-access-988rz\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.986854 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.986582 5959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-envoy\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.986854 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:21.986590 5959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e-istio-podinfo\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:22.065955 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.065923 5959 generic.go:356] "Generic (PLEG): container finished" podID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerID="88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b" exitCode=0 Oct 30 19:18:22.066131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.065977 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" Oct 30 19:18:22.066131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.066012 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" event={"ID":"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e","Type":"ContainerDied","Data":"88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b"} Oct 30 19:18:22.066131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.066070 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk" event={"ID":"5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e","Type":"ContainerDied","Data":"d5ef03712c7b05bbe6da6b94621b54ba3f46b8ed4e140464296126a079dbab25"} Oct 30 19:18:22.066131 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.066099 5959 scope.go:122] "RemoveContainer" containerID="88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b" Oct 30 19:18:22.075511 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.075493 5959 scope.go:122] "RemoveContainer" containerID="88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b" Oct 30 19:18:22.075821 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:18:22.075802 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b\": container with ID starting with 88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b not found: ID does not exist" containerID="88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b" Oct 30 19:18:22.075898 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.075829 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b"} err="failed to get container status \"88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b\": rpc error: code = NotFound desc = could not find container \"88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b\": container with ID starting with 88d9cfcea33873576fc846273b583a0705679fa565c552e84dadf9caf9f19a4b not found: ID does not exist" Oct 30 19:18:22.088063 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.088041 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk"] Oct 30 19:18:22.098175 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.098152 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-kjkjk"] Oct 30 19:18:22.442196 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:22.442167 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-crll9_a5f538d4-ba0c-4f64-953d-5f779027b94c/kuadrant-console-plugin/0.log" Oct 30 19:18:23.049156 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:18:23.049130 5959 logging.go:55] [core] [Channel #5811 SubChannel #5812]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.50:50052: connect: connection refused" Oct 30 19:18:23.255023 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.254998 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 19:18:23.296779 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.296689 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725eb7d0-4077-4c1d-9a15-158e18f92fdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725eb7d0-4077-4c1d-9a15-158e18f92fdb-plugins-config-volume\") pod \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\" (UID: \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\") " Oct 30 19:18:23.296779 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.296731 5959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725eb7d0-4077-4c1d-9a15-158e18f92fdb-kube-api-access-rs8qf\" (UniqueName: \"kubernetes.io/projected/725eb7d0-4077-4c1d-9a15-158e18f92fdb-kube-api-access-rs8qf\") pod \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\" (UID: \"725eb7d0-4077-4c1d-9a15-158e18f92fdb\") " Oct 30 19:18:23.297077 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.297050 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725eb7d0-4077-4c1d-9a15-158e18f92fdb-plugins-config-volume" pod "725eb7d0-4077-4c1d-9a15-158e18f92fdb" (UID: "725eb7d0-4077-4c1d-9a15-158e18f92fdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:23.299000 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.298974 5959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725eb7d0-4077-4c1d-9a15-158e18f92fdb-kube-api-access-rs8qf" pod "725eb7d0-4077-4c1d-9a15-158e18f92fdb" (UID: "725eb7d0-4077-4c1d-9a15-158e18f92fdb"). InnerVolumeSpecName "kube-api-access-rs8qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:23.398342 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.398301 5959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725eb7d0-4077-4c1d-9a15-158e18f92fdb-plugins-config-volume\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:23.398342 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.398334 5959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs8qf\" (UniqueName: \"kubernetes.io/projected/725eb7d0-4077-4c1d-9a15-158e18f92fdb-kube-api-access-rs8qf\") on node \"ip-10-0-133-96.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:23.783720 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.783691 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d9fgc_bb1241e0-96c3-480d-b527-cd2662e82ef2/limitador/0.log" Oct 30 19:18:23.795512 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:23.795483 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" path="/var/lib/kubelet/pods/5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e/volumes" Oct 30 19:18:24.050008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.049911 5959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" podUID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.50:50052\" within 1s: context deadline exceeded" Oct 30 19:18:24.050008 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.050001 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 19:18:24.050486 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:18:24.050000 5959 logging.go:55] [core] [Channel #5811 SubChannel #5812]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.50:50052: operation was canceled" Oct 30 19:18:24.077079 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.077040 5959 generic.go:356] "Generic (PLEG): container finished" podID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" containerID="bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab" exitCode=0 Oct 30 19:18:24.077236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.077090 5959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" Oct 30 19:18:24.077236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.077125 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" event={"ID":"725eb7d0-4077-4c1d-9a15-158e18f92fdb","Type":"ContainerDied","Data":"bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab"} Oct 30 19:18:24.077236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.077182 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x" event={"ID":"725eb7d0-4077-4c1d-9a15-158e18f92fdb","Type":"ContainerDied","Data":"cf0be9ba43b6e616be32c31592fcc92bbe58b98a5e1a0ad08aa115b92c17018d"} Oct 30 19:18:24.077236 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.077204 5959 scope.go:122] "RemoveContainer" containerID="bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab" Oct 30 19:18:24.086099 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.086080 5959 scope.go:122] "RemoveContainer" containerID="bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab" Oct 30 19:18:24.086386 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:18:24.086366 5959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab\": container with ID starting with bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab not found: ID does not exist" containerID="bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab" Oct 30 19:18:24.086459 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.086392 5959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab"} err="failed to get container status \"bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab\": rpc error: code = NotFound desc = could not find container \"bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab\": container with ID starting with bd9bbcf31d7d6ebbc25d69484dfda6517fc883f9d490118e9cf2187fef6d7cab not found: ID does not exist" Oct 30 19:18:24.093840 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.093816 5959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x"] Oct 30 19:18:24.097794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:24.097774 5959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85cb41-68854975f5-xlf6x"] Oct 30 19:18:25.297446 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:25.297397 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7gnzw_1e5b9831-369a-4cbd-b808-0e3dca949a09/manager/0.log" Oct 30 19:18:25.794824 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:25.794795 5959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" path="/var/lib/kubelet/pods/725eb7d0-4077-4c1d-9a15-158e18f92fdb/volumes" Oct 30 19:18:26.120873 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:26.120775 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zjvvq_523e3b72-74f3-4826-b00e-c0c7688620c4/server/0.log" Oct 30 19:18:27.778768 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:18:27.778732 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:18:38.321437 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:38.321398 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7gnzw_1e5b9831-369a-4cbd-b808-0e3dca949a09/manager/0.log" Oct 30 19:18:39.117175 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:39.117149 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zjvvq_523e3b72-74f3-4826-b00e-c0c7688620c4/server/0.log" Oct 30 19:18:44.905504 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:44.905469 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-crll9_a5f538d4-ba0c-4f64-953d-5f779027b94c/kuadrant-console-plugin/0.log" Oct 30 19:18:45.900195 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:45.900165 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d9fgc_bb1241e0-96c3-480d-b527-cd2662e82ef2/limitador/0.log" Oct 30 19:18:47.038836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:47.038808 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gs6dj_5807139c-f7d5-4f02-a866-88a6cb783af9/discovery/0.log" Oct 30 19:18:47.990010 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:47.989980 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f89dc687f-h9g5s_a6f75d54-cda6-4732-aedd-1250c4aa2c57/kube-auth-proxy/0.log" Oct 30 19:18:48.351546 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:48.351456 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-68cb9b4b95-bkpg7_bf0d5ca3-7ba8-481c-b29d-702244848bbf/istio-proxy/0.log" Oct 30 19:18:48.694000 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:18:48.693969 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-68cb9b4b95-q656v_52b663c4-e4ce-433f-80b6-52fe26e9f976/istio-proxy/0.log" Oct 30 19:18:57.778956 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:18:57.778910 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:19:02.171776 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:02.171746 5959 scope.go:122] "RemoveContainer" containerID="65454e5369e4ac7b39703a1bdc8be3c9e1b3cd4f224b0217dd774a6175bd86f9" Oct 30 19:19:02.180180 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:02.180159 5959 scope.go:122] "RemoveContainer" containerID="a5e1f7e45471b8664c5da68a58c5840cceba8d5131941dfb447f0641069841f1" Oct 30 19:19:07.646948 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:07.646905 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-628cx_7d77554d-8094-4546-8fe7-0edf66752a5e/global-pull-secret-syncer/0.log" Oct 30 19:19:08.072011 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:08.071920 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g6tw4_ce5ba25c-da20-40d5-b02f-89e94aaad0d2/konnectivity-agent/0.log" Oct 30 19:19:08.097306 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:08.097267 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-96.us-east-2.compute.internal_e8517d60336db91a5d4c1afa2b83dc77/haproxy/0.log" Oct 30 19:19:15.800349 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:15.800244 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-crll9_a5f538d4-ba0c-4f64-953d-5f779027b94c/kuadrant-console-plugin/0.log" Oct 30 19:19:17.031554 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:17.031518 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-d9fgc_bb1241e0-96c3-480d-b527-cd2662e82ef2/limitador/0.log" Oct 30 19:19:18.603878 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.603831 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_641f46c9-7931-4981-89ab-3f056fed93e1/alertmanager/0.log" Oct 30 19:19:18.627792 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.627764 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_641f46c9-7931-4981-89ab-3f056fed93e1/config-reloader/0.log" Oct 30 19:19:18.650043 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.649996 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_641f46c9-7931-4981-89ab-3f056fed93e1/kube-rbac-proxy-web/0.log" Oct 30 19:19:18.673914 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.673888 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_641f46c9-7931-4981-89ab-3f056fed93e1/kube-rbac-proxy/0.log" Oct 30 19:19:18.697959 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.697932 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_641f46c9-7931-4981-89ab-3f056fed93e1/kube-rbac-proxy-metric/0.log" Oct 30 19:19:18.720583 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.720555 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_641f46c9-7931-4981-89ab-3f056fed93e1/prom-label-proxy/0.log" Oct 30 19:19:18.743593 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.743577 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_641f46c9-7931-4981-89ab-3f056fed93e1/init-config-reloader/0.log" Oct 30 19:19:18.888427 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.888405 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cmsmg_d8e52eca-6d99-4f5c-bbee-9f9559130add/kube-state-metrics/0.log" Oct 30 19:19:18.907590 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.907565 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cmsmg_d8e52eca-6d99-4f5c-bbee-9f9559130add/kube-rbac-proxy-main/0.log" Oct 30 19:19:18.932639 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.932596 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cmsmg_d8e52eca-6d99-4f5c-bbee-9f9559130add/kube-rbac-proxy-self/0.log" Oct 30 19:19:18.963581 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:18.963551 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c7bcfc979-g65nx_d3e4cb6c-9268-4dda-a3a8-c8bb503942fb/metrics-server/0.log" Oct 30 19:19:19.104522 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.104496 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrqjz_73bfd18d-99d5-4967-b748-00d94f5dc398/node-exporter/0.log" Oct 30 19:19:19.132794 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.132769 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrqjz_73bfd18d-99d5-4967-b748-00d94f5dc398/kube-rbac-proxy/0.log" Oct 30 19:19:19.158778 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.158715 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrqjz_73bfd18d-99d5-4967-b748-00d94f5dc398/init-textfile/0.log" Oct 30 19:19:19.267983 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.267959 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qpm5s_9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/kube-rbac-proxy-main/0.log" Oct 30 19:19:19.291836 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.291808 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qpm5s_9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/kube-rbac-proxy-self/0.log" Oct 30 19:19:19.318001 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.317977 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qpm5s_9bb2bc4b-5c95-469f-bdca-5fe3862f23ad/openshift-state-metrics/0.log" Oct 30 19:19:19.358971 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.358938 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f096784b-2749-4f7e-adb3-448300f1013c/prometheus/0.log" Oct 30 19:19:19.382924 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.382897 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f096784b-2749-4f7e-adb3-448300f1013c/config-reloader/0.log" Oct 30 19:19:19.411048 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.410975 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f096784b-2749-4f7e-adb3-448300f1013c/thanos-sidecar/0.log" Oct 30 19:19:19.438879 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.438846 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f096784b-2749-4f7e-adb3-448300f1013c/kube-rbac-proxy-web/0.log" Oct 30 19:19:19.466747 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.466704 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f096784b-2749-4f7e-adb3-448300f1013c/kube-rbac-proxy/0.log" Oct 30 19:19:19.490288 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.490259 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f096784b-2749-4f7e-adb3-448300f1013c/kube-rbac-proxy-thanos/0.log" Oct 30 19:19:19.520684 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.520662 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f096784b-2749-4f7e-adb3-448300f1013c/init-config-reloader/0.log" Oct 30 19:19:19.554275 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.554242 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-mz6n4_e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/prometheus-operator/0.log" Oct 30 19:19:19.575200 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.575174 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-mz6n4_e0ef9c9a-e45e-447f-b68c-56ef8d59ab96/kube-rbac-proxy/0.log" Oct 30 19:19:19.634344 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.634319 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5784486569-4gvpd_d144b776-a2c9-450c-b189-e299d6d447f6/telemeter-client/0.log" Oct 30 19:19:19.661518 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.661449 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5784486569-4gvpd_d144b776-a2c9-450c-b189-e299d6d447f6/reload/0.log" Oct 30 19:19:19.687323 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.687299 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5784486569-4gvpd_d144b776-a2c9-450c-b189-e299d6d447f6/kube-rbac-proxy/0.log" Oct 30 19:19:19.729804 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.729780 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd85567c4-mkwgs_a5fc82ca-c9cf-43f1-871c-c156ea050ca6/thanos-query/0.log" Oct 30 19:19:19.753239 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.753216 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd85567c4-mkwgs_a5fc82ca-c9cf-43f1-871c-c156ea050ca6/kube-rbac-proxy-web/0.log" Oct 30 19:19:19.785084 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.785052 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd85567c4-mkwgs_a5fc82ca-c9cf-43f1-871c-c156ea050ca6/kube-rbac-proxy/0.log" Oct 30 19:19:19.809373 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.809352 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd85567c4-mkwgs_a5fc82ca-c9cf-43f1-871c-c156ea050ca6/prom-label-proxy/0.log" Oct 30 19:19:19.834197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.834175 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd85567c4-mkwgs_a5fc82ca-c9cf-43f1-871c-c156ea050ca6/kube-rbac-proxy-rules/0.log" Oct 30 19:19:19.857277 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.857261 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd85567c4-mkwgs_a5fc82ca-c9cf-43f1-871c-c156ea050ca6/kube-rbac-proxy-metrics/0.log" Oct 30 19:19:19.913787 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.913719 5959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw"] Oct 30 19:19:19.914128 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914113 5959 cpu_manager.go:414] "containerMap: removing container" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" Oct 30 19:19:19.914180 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914131 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" Oct 30 19:19:19.914180 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914148 5959 cpu_manager.go:414] "containerMap: removing container" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" Oct 30 19:19:19.914180 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914153 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" Oct 30 19:19:19.914180 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914162 5959 cpu_manager.go:414] "containerMap: removing container" podUID="f126e5dc-9d83-4e52-8c42-ca8390a07330" containerName="payload-pre-processing" Oct 30 19:19:19.914180 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914167 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f126e5dc-9d83-4e52-8c42-ca8390a07330" containerName="payload-pre-processing" Oct 30 19:19:19.914180 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914175 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" Oct 30 19:19:19.914180 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914179 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914185 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914190 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914195 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" containerName="cleanup" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914199 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" containerName="cleanup" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914209 5959 cpu_manager.go:414] "containerMap: removing container" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914215 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914231 5959 cpu_manager.go:414] "containerMap: removing container" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerName="cleanup" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914235 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerName="cleanup" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914244 5959 cpu_manager.go:414] "containerMap: removing container" podUID="7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" containerName="payload-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914247 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" containerName="payload-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914253 5959 cpu_manager.go:414] "containerMap: removing container" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914256 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914263 5959 cpu_manager.go:414] "containerMap: removing container" podUID="a1c0c547-d980-42ee-8b82-411a2bf05e64" containerName="maas-api" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914267 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c0c547-d980-42ee-8b82-411a2bf05e64" containerName="maas-api" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914273 5959 cpu_manager.go:414] "containerMap: removing container" podUID="1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" containerName="payload-pre-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914277 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" containerName="payload-pre-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914284 5959 cpu_manager.go:414] "containerMap: removing container" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914288 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914304 5959 cpu_manager.go:414] "containerMap: removing container" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914310 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914317 5959 cpu_manager.go:414] "containerMap: removing container" podUID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" containerName="payload-pre-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914321 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" containerName="payload-pre-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914329 5959 cpu_manager.go:414] "containerMap: removing container" podUID="21ebf898-38d1-4786-8ebc-d5e53a52961f" containerName="payload-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914333 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ebf898-38d1-4786-8ebc-d5e53a52961f" containerName="payload-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914339 5959 cpu_manager.go:414] "containerMap: removing container" podUID="122408a8-f58c-46bf-b236-5beaf8afd104" containerName="maas-api" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914343 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122408a8-f58c-46bf-b236-5beaf8afd104" containerName="maas-api" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914351 5959 cpu_manager.go:414] "containerMap: removing container" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914355 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914363 5959 cpu_manager.go:414] "containerMap: removing container" podUID="424643ce-2ec3-462d-82a8-f5a61ea7a329" containerName="maas-api" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914369 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424643ce-2ec3-462d-82a8-f5a61ea7a329" containerName="maas-api" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914380 5959 cpu_manager.go:414] "containerMap: removing container" podUID="f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" containerName="payload-pre-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914387 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" containerName="payload-pre-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914393 5959 cpu_manager.go:414] "containerMap: removing container" podUID="e4db960a-a125-48a3-a095-0cbc7d6b79b3" containerName="payload-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914398 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4db960a-a125-48a3-a095-0cbc7d6b79b3" containerName="payload-processing" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914405 5959 cpu_manager.go:414] "containerMap: removing container" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" Oct 30 19:19:19.914398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914410 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914417 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914421 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914427 5959 cpu_manager.go:414] "containerMap: removing container" podUID="cff7c99c-0d72-4542-a2ef-28910e83d0bf" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914430 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff7c99c-0d72-4542-a2ef-28910e83d0bf" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914436 5959 cpu_manager.go:414] "containerMap: removing container" podUID="7676fec7-190b-4c60-8cbd-aded416ed747" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914440 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7676fec7-190b-4c60-8cbd-aded416ed747" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914446 5959 cpu_manager.go:414] "containerMap: removing container" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914450 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914461 5959 cpu_manager.go:414] "containerMap: removing container" podUID="108e4e52-68b1-458e-aca1-8706621e3a57" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914467 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108e4e52-68b1-458e-aca1-8706621e3a57" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914476 5959 cpu_manager.go:414] "containerMap: removing container" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914480 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914484 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ddb38225-6af5-4a9a-824a-2e232854b71a" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914489 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb38225-6af5-4a9a-824a-2e232854b71a" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914494 5959 cpu_manager.go:414] "containerMap: removing container" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914498 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914502 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914506 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914514 5959 cpu_manager.go:414] "containerMap: removing container" podUID="d0da26a2-90ab-43e0-9476-bcb1620532b8" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914518 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0da26a2-90ab-43e0-9476-bcb1620532b8" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914523 5959 cpu_manager.go:414] "containerMap: removing container" podUID="89938e5d-7f44-484a-ab58-355bc498e717" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914527 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89938e5d-7f44-484a-ab58-355bc498e717" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914536 5959 cpu_manager.go:414] "containerMap: removing container" podUID="74f87578-e8f2-4a8a-9e6b-0525fd6c45db" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914543 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f87578-e8f2-4a8a-9e6b-0525fd6c45db" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914553 5959 cpu_manager.go:414] "containerMap: removing container" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914557 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914563 5959 cpu_manager.go:414] "containerMap: removing container" podUID="f2581e0a-b5b3-4df8-8b36-7f9b5232addc" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914566 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2581e0a-b5b3-4df8-8b36-7f9b5232addc" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914571 5959 cpu_manager.go:414] "containerMap: removing container" podUID="49ce5351-e16e-498b-a06d-8a70ca71768b" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914575 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ce5351-e16e-498b-a06d-8a70ca71768b" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914582 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914586 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914592 5959 cpu_manager.go:414] "containerMap: removing container" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914597 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914619 5959 cpu_manager.go:414] "containerMap: removing container" podUID="23bd45ad-4844-4771-ab04-be34f44f15f2" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914627 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23bd45ad-4844-4771-ab04-be34f44f15f2" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914637 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914641 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914651 5959 cpu_manager.go:414] "containerMap: removing container" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914655 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914660 5959 cpu_manager.go:414] "containerMap: removing container" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914665 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914671 5959 cpu_manager.go:414] "containerMap: removing container" podUID="1bb68f2a-2ebc-4def-91d1-c8258f64dc84" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914675 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb68f2a-2ebc-4def-91d1-c8258f64dc84" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914681 5959 cpu_manager.go:414] "containerMap: removing container" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914685 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914692 5959 cpu_manager.go:414] "containerMap: removing container" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914699 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914708 5959 cpu_manager.go:414] "containerMap: removing container" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914714 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914719 5959 cpu_manager.go:414] "containerMap: removing container" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914724 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914729 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914733 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914741 5959 cpu_manager.go:414] "containerMap: removing container" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914744 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914751 5959 cpu_manager.go:414] "containerMap: removing container" podUID="036f65e0-fe98-4601-b469-3c033588c81f" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914755 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036f65e0-fe98-4601-b469-3c033588c81f" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914761 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914765 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914770 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914775 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914786 5959 cpu_manager.go:414] "containerMap: removing container" podUID="5eb94bfc-8707-4375-9f7f-5c454fcb06e2" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914792 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb94bfc-8707-4375-9f7f-5c454fcb06e2" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914800 5959 cpu_manager.go:414] "containerMap: removing container" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914807 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914816 5959 cpu_manager.go:414] "containerMap: removing container" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914820 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914825 5959 cpu_manager.go:414] "containerMap: removing container" podUID="cb7d1645-e7a7-433a-a39f-8d53f47d01ef" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914828 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7d1645-e7a7-433a-a39f-8d53f47d01ef" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914834 5959 cpu_manager.go:414] "containerMap: removing container" podUID="cd779df3-489c-491c-8f35-747a9536e4dd" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914838 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd779df3-489c-491c-8f35-747a9536e4dd" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914845 5959 cpu_manager.go:414] "containerMap: removing container" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914849 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914855 5959 cpu_manager.go:414] "containerMap: removing container" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914859 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914865 5959 cpu_manager.go:414] "containerMap: removing container" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914872 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914883 5959 cpu_manager.go:414] "containerMap: removing container" podUID="8de79c7d-5fbf-4bd1-9916-f696c5a07901" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914890 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de79c7d-5fbf-4bd1-9916-f696c5a07901" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914896 5959 cpu_manager.go:414] "containerMap: removing container" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914900 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914907 5959 cpu_manager.go:414] "containerMap: removing container" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914911 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914917 5959 cpu_manager.go:414] "containerMap: removing container" podUID="e9633de6-8e2a-45ee-a960-5c6e797c384b" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914934 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9633de6-8e2a-45ee-a960-5c6e797c384b" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914941 5959 cpu_manager.go:414] "containerMap: removing container" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914946 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914951 5959 cpu_manager.go:414] "containerMap: removing container" podUID="f1c4c0b9-5214-4046-8ce2-ef1da3161f17" containerName="revoke-keys" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914956 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c4c0b9-5214-4046-8ce2-ef1da3161f17" containerName="revoke-keys" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914963 5959 cpu_manager.go:414] "containerMap: removing container" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914967 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914972 5959 cpu_manager.go:414] "containerMap: removing container" podUID="0aa10346-b095-4bcb-83c9-b09741af930d" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.914976 5959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa10346-b095-4bcb-83c9-b09741af930d" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915043 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915051 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0d64dc-513a-4110-9364-29c7960c74a3" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915057 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ae7e50-6451-4b7e-8691-999aa3234d94" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915061 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb68f2a-2ebc-4def-91d1-c8258f64dc84" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915067 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edfaee8-04db-4f64-bdec-efe3df8a5439" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915072 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab751835-0c83-473f-a8ff-b17e93a9a3a3" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915079 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="03100ce6-ad8c-4e96-8403-1da2674f46b5" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915085 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7b9b00-3bc3-4ece-8926-41355d0a9b5e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915092 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915097 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cde4d2c-1782-41cc-9011-a7ac2ee0d079" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915102 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="036f65e0-fe98-4601-b469-3c033588c81f" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915107 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0c1081-4209-4c4b-912c-e1defc785003" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915112 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae6438c-e0eb-4ac2-b04f-6e61c6bb504e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915116 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb94bfc-8707-4375-9f7f-5c454fcb06e2" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915121 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fa02a0-22e8-4b53-924b-19e6cb605d5b" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915125 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea10d0b-617b-4aa0-a73d-5d22a0e7153a" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915130 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7d1645-e7a7-433a-a39f-8d53f47d01ef" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915136 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd779df3-489c-491c-8f35-747a9536e4dd" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915143 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb88e37-7ad7-4e05-b2af-f2571e01a2d2" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915147 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e0824e-05df-42c1-a737-90283fecef74" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915153 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d5d58c-3291-49e5-9e98-e2989a67316a" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915157 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de79c7d-5fbf-4bd1-9916-f696c5a07901" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915162 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3860e2-becb-4660-9d93-ff8c50373fb7" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915167 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915171 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a468ed9d-e51f-422b-927d-abce4d0c7a36" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915176 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9633de6-8e2a-45ee-a960-5c6e797c384b" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915188 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="78abee8b-e5f9-4ff4-be79-ca766fbb26c0" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915193 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c4c0b9-5214-4046-8ce2-ef1da3161f17" containerName="revoke-keys" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915199 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e4ec7d-7b73-42cc-94d3-5eb46d09872f" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915204 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa10346-b095-4bcb-83c9-b09741af930d" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915210 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cc2519-050e-4689-975a-b33a9dd76d5e" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915234 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb53f1d-da29-4ce3-bd69-19aa0c1dd873" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915239 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f126e5dc-9d83-4e52-8c42-ca8390a07330" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915243 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8563f5fd-0739-44cb-acbf-e98721991c05" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915248 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea59e1b2-c9a5-4322-83f5-e8c1d9777880" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915253 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2606db6-8b96-4360-9519-6cda6ca1380f" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915260 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fef6d6c-ffa6-4ce8-a406-e4265da4f9d7" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915264 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd33ce2-06d2-4680-af9a-58e00ea3ac03" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915271 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b41146-9abf-4010-8c86-c473bbf18e19" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915276 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c0c547-d980-42ee-8b82-411a2bf05e64" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915281 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7b40fc-9360-4a4b-baa0-02fcb9063ac0" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915287 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cae719-52c3-492a-aa66-c4db3a011fee" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915293 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ba3eb5-1b12-4e2b-9fec-9842167e514a" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915299 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="725eb7d0-4077-4c1d-9a15-158e18f92fdb" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915304 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ebf898-38d1-4786-8ebc-d5e53a52961f" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915308 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="122408a8-f58c-46bf-b236-5beaf8afd104" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915313 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98f9d6d-0773-4ae0-9a6b-b9aa7c880a07" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915319 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="424643ce-2ec3-462d-82a8-f5a61ea7a329" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915324 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4db960a-a125-48a3-a095-0cbc7d6b79b3" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915329 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="359be520-f69e-4d59-8a67-55e6a11236a5" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915335 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ceedfeb-9369-4ad4-bb4a-d23c669a6e34" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915340 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff7c99c-0d72-4542-a2ef-28910e83d0bf" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915346 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7676fec7-190b-4c60-8cbd-aded416ed747" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915351 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61ed8c9-8ea9-4af5-a148-51a1b3b790e4" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915355 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="108e4e52-68b1-458e-aca1-8706621e3a57" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915362 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9ce7f1-3bff-4808-ab73-d1281dd72f98" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915367 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb38225-6af5-4a9a-824a-2e232854b71a" containerName="payload-pre-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915372 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c654a045-0394-4c1f-a8fe-a67f9e22ea2e" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915377 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0da26a2-90ab-43e0-9476-bcb1620532b8" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915382 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="89938e5d-7f44-484a-ab58-355bc498e717" containerName="maas-api" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915388 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f87578-e8f2-4a8a-9e6b-0525fd6c45db" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915392 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e300afa7-3bad-4988-a244-2c0dfb446993" containerName="main" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915398 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2581e0a-b5b3-4df8-8b36-7f9b5232addc" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915403 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ce5351-e16e-498b-a06d-8a70ca71768b" containerName="payload-processing" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915409 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef922752-ebe2-4428-81b5-95e146bff23f" containerName="cleanup" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915414 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a43000-522d-4ea3-9d38-4ceaa5789aea" containerName="istio-proxy" Oct 30 19:19:19.915665 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.915419 5959 memory_manager.go:365] "RemoveStaleState removing state" podUID="23bd45ad-4844-4771-ab04-be34f44f15f2" containerName="payload-pre-processing" Oct 30 19:19:19.920747 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.920155 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:19.926127 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.926103 5959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47xdf\"/\"default-dockercfg-xpwfn\"" Oct 30 19:19:19.926641 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.926597 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47xdf\"/\"openshift-service-ca.crt\"" Oct 30 19:19:19.927401 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.927383 5959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47xdf\"/\"kube-root-ca.crt\"" Oct 30 19:19:19.927982 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.927862 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw"] Oct 30 19:19:19.973474 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.973440 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdb88\" (UniqueName: \"kubernetes.io/projected/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-kube-api-access-jdb88\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:19.973676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.973494 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-lib-modules\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:19.973676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.973575 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-sys\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:19.973676 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.973642 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-proc\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:19.973791 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:19.973689 5959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-podres\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074442 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074414 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-sys\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074454 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-proc\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074474 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-podres\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074551 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-sys\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074558 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-proc\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074634 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074575 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdb88\" (UniqueName: \"kubernetes.io/projected/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-kube-api-access-jdb88\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074819 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074638 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-podres\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074819 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074642 5959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-lib-modules\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.074819 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.074707 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-lib-modules\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.082717 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.082687 5959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdb88\" (UniqueName: \"kubernetes.io/projected/999d174f-fc8b-44ba-98e9-f8c0174f6b5d-kube-api-access-jdb88\") pod \"perf-node-gather-daemonset-cqzmw\" (UID: \"999d174f-fc8b-44ba-98e9-f8c0174f6b5d\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.233567 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.233507 5959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:20.509351 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:20.509265 5959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw"] Oct 30 19:19:20.512651 ip-10-0-133-96 kubenswrapper[5959]: W1030 19:19:20.512595 5959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod999d174f_fc8b_44ba_98e9_f8c0174f6b5d.slice/crio-dc64531fd7a41819d6311d48901b09f6866d468e748d32c5f94b6b5fdfae4605 WatchSource:0}: Error finding container dc64531fd7a41819d6311d48901b09f6866d468e748d32c5f94b6b5fdfae4605: Status 404 returned error can't find the container with id dc64531fd7a41819d6311d48901b09f6866d468e748d32c5f94b6b5fdfae4605 Oct 30 19:19:21.344398 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:21.344354 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" event={"ID":"999d174f-fc8b-44ba-98e9-f8c0174f6b5d","Type":"ContainerStarted","Data":"609f90145c0804cd83b486b6017e738e579a55bb8d984734153f9a8b8336524b"} Oct 30 19:19:21.344840 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:21.344451 5959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" event={"ID":"999d174f-fc8b-44ba-98e9-f8c0174f6b5d","Type":"ContainerStarted","Data":"dc64531fd7a41819d6311d48901b09f6866d468e748d32c5f94b6b5fdfae4605"} Oct 30 19:19:21.344840 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:21.344478 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:21.362090 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:21.362029 5959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" podStartSLOduration=2.362011929 podStartE2EDuration="2.362011929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:21.361589993 +0000 UTC m=+2541.690001377" watchObservedRunningTime="2026-10-30 19:19:21.362011929 +0000 UTC m=+2541.690423314" Oct 30 19:19:22.927619 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:22.927588 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67577fc95c-4n59h_65da6205-b5cb-423f-b270-5122dd5b502b/console/0.log" Oct 30 19:19:25.440468 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:25.440428 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7gnzw_1e5b9831-369a-4cbd-b808-0e3dca949a09/manager/0.log" Oct 30 19:19:25.734197 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:25.734120 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7gnzw_1e5b9831-369a-4cbd-b808-0e3dca949a09/manager/0.log" Oct 30 19:19:26.058833 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:26.058761 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45mzz_8d86b7d9-2e05-4f95-b002-931249ab66e2/dns/0.log" Oct 30 19:19:26.282528 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:26.282507 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zjvvq_523e3b72-74f3-4826-b00e-c0c7688620c4/server/0.log" Oct 30 19:19:26.286069 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:26.286052 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45mzz_8d86b7d9-2e05-4f95-b002-931249ab66e2/kube-rbac-proxy/0.log" Oct 30 19:19:26.489440 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:26.489408 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zjvvq_523e3b72-74f3-4826-b00e-c0c7688620c4/server/0.log" Oct 30 19:19:26.548068 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:26.548042 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-msdfn_c8994e33-9b53-41d7-8fa8-04463a3ae647/dns-node-resolver/0.log" Oct 30 19:19:27.359798 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:27.359770 5959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-cqzmw" Oct 30 19:19:27.554914 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:27.554879 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-slltd_272dcf18-8186-45d9-9316-8c5a0efc951b/node-ca/0.log" Oct 30 19:19:27.777692 ip-10-0-133-96 kubenswrapper[5959]: E1030 19:19:27.777658 5959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbqcr" containerName="extractor" Oct 30 19:19:28.667115 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:28.667075 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7gnzw_1e5b9831-369a-4cbd-b808-0e3dca949a09/manager/0.log" Oct 30 19:19:28.896900 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:28.896871 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7gnzw_1e5b9831-369a-4cbd-b808-0e3dca949a09/manager/0.log" Oct 30 19:19:29.564472 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:29.564447 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zjvvq_523e3b72-74f3-4826-b00e-c0c7688620c4/server/0.log" Oct 30 19:19:29.836193 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:29.836108 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zjvvq_523e3b72-74f3-4826-b00e-c0c7688620c4/server/0.log" Oct 30 19:19:30.339418 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:30.339382 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gs6dj_5807139c-f7d5-4f02-a866-88a6cb783af9/discovery/0.log" Oct 30 19:19:32.529632 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:32.529574 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7gnzw_1e5b9831-369a-4cbd-b808-0e3dca949a09/manager/0.log" Oct 30 19:19:32.703521 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:32.703493 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7gnzw_1e5b9831-369a-4cbd-b808-0e3dca949a09/manager/0.log" Oct 30 19:19:33.689260 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:33.689227 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zjvvq_523e3b72-74f3-4826-b00e-c0c7688620c4/server/0.log" Oct 30 19:19:33.999987 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:33.999889 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zjvvq_523e3b72-74f3-4826-b00e-c0c7688620c4/server/0.log" Oct 30 19:19:35.165190 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:35.165144 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gs6dj_5807139c-f7d5-4f02-a866-88a6cb783af9/discovery/0.log" Oct 30 19:19:37.216216 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:37.216148 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gs6dj_5807139c-f7d5-4f02-a866-88a6cb783af9/discovery/0.log" Oct 30 19:19:37.545473 ip-10-0-133-96 kubenswrapper[5959]: I1030 19:19:37.545392 5959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gs6dj_5807139c-f7d5-4f02-a866-88a6cb783af9/discovery/0.log"